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Abstract 

Despite  numerous  reform  efforts  over  the  past  30  years,  acquisition  programs  in 
the  Department  of  Defense  (DoD)  continue  to  experience  cost  overruns  and  schedule 
delays.  One  contributing  factor  is  the  decision-making  process  used  by  defense  officials. 
The  General  Accounting  Office  (GAO)  has  stated  that  ‘poor  program  outcomes  are  the 
lack  of  widespread  adoption  of  a  knowledge-based  acquisition  process  within  DoD 
despite  polices  that  support  such  a  process.  A  knowledge-based  business  case  at  the 
outset  of  each  program  would  alleviate  overpromising  on  cost,  schedule,  and 
performance  and  would  empower  program  managers.’ 

Effective  decision-making  for  acquisition  programs  is  very  important.  It  not  only 
affects  the  performance  of  a  program  but  could  also  impact  the  lives  of  Airman,  Soldiers, 
Sailors,  and  Marines  protecting  our  country.  Analyzing  decision  support  products  is  one 
method  to  improve  the  knowledge  used  during  the  decision-making  process.  Therefore, 
the  scope  of  this  research  focused  on  knowledge  products  supporting  decisions  made  by 
DoD  acquisition  officials  and  their  alignment  with  best  practices  and  their  usefulness  to 
decision-makers. 

This  research  found  that  the  required  information  contained  in  decision  support 
products  is  not  adequate  to  provide  the  knowledge  needed  to  make  informed  decisions. 
Recommendations  for  improving  decision  support  for  key  knowledge  areas  will  be 
discussed. 
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KNOWLEDGE-BASED  DECISION  SUPPORT  IN  DOD  ACQUISITIONS 


Chapter  I.  Introduction 

Changes  in  the  acquisition  process  in  the  Department  of  Defense  (DoD)  have 
been  on-going  over  the  course  of  the  past  three  decades.  The  problem,  however,  is  that 
most  reforms  that  have  been  implemented  have  largely  been  unsuccessful  (Friedman, 
2009).  The  reasons  for  the  failure  of  acquisition  reform  are  complicated  and  are  due  to 
many  different  factors.  Despite  these  reform  efforts,  many  programs  are  still  being 
completed  behind  schedule  and  with  substantial  cost  overruns.  Even  worse,  the 
acquisition  process  within  the  DoD  has  actually  become  less  consistent  (Dawn  et  al., 
1998).  The  end-result  is  an  acquisition  system  that  is  facing  greater  problems  and 
resulting  in  even  more  wasted  resources  for  the  United  States  military  (Gill,  2001). 

One  problem  that  exists  for  the  DoD  is  that  the  acquisition  process  is  largely 
based  on  external  influences  as  opposed  to  a  genuine  knowledge-based  approach  to 
acquiring  new  systems  and  materials  (Ignols  and  Brem,  1998).  What  is  needed  for  the 
DoD  acquisition  community  is  to  ensure  that  the  right  knowledge  is  captured  at  key 
decision  points  to  determine  the  most  effective  means  by  which  to  acquire  new  weapon 
systems.  Therefore,  this  research  is  designed  to  investigate  what  is  occurring  within  the 
DoD  acquisition  decision-making  process.  It  is  these  decisions  (most  of  which  are  made 
early  in  a  program’s  life-cycle)  that  are  causing  less  than  ideal  results  with  regard  to 
weapon  system  programs  being  behind  schedule,  over  budget,  and  at  times  not 
performing  as  planned. 
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Background 

In  its  most  basic  form,  knowledge-based  decision-making  is  simply  the  process  of 
collecting  and  using  data  and  information  to  gain  knowledge  that  can  be  used  to  make 
informed  decisions  (Fay,  2007).  This  process  can  be  a  powerful  tool  in  which 
organizations  utilize  the  synthesis  of  data  and  information  that  will  lead  to  making  better 
decisions.  Although  used  by  both  public  and  private  sector  organizations,  there  are 
differences  in  the  ways  in  which  they  utilize  this  process.  The  differences  are  normally 
attributed  to  the  role  that  these  organizations  play  in  society.  However,  although 
recognized,  the  influence  of  the  context  in  which  knowledge-based  decision  support  is 
made  is  largely  unexplored  (Papadakis  and  Barwise,  1998). 

In  the  private  sector,  through  market  research,  companies  use  knowledge-based 
decision  support  primarily  to  make  predictions  about  customer  behaviors  and 
expectations.  They  then  attempt  to  meet  those  behaviors  and  expectations  with  products 
and  services.  Companies  in  the  private  sector  collect  large  amounts  of  data  about  their 
customers  and  even  the  customers  of  competing  companies  to  understand  their  needs  and 
desires  (Doukidis,  Mylonopoulos  &  Pouloudi,  2004).  In  other  words,  most  companies  in 
the  private  sector  do  not  make  decisions  until  they  have  collected  large  amounts  of  data 
about  the  people  they  serve.  Then,  the  decisions  made  by  everyone  from  company 
executives  to  front-line  personnel  are  based  on  the  knowledge  that  is  gained  from 
analyzing  the  collected  data  (Vercellis,  2009). 

For  the  private  sector,  an  entire  industry  exists  of  companies  that  do  nothing  more 
than  collect  and  analyze  data  about  customer  actions,  behaviors,  and  attitudes. 
Corporations  in  the  private  sector  typically  only  make  decisions  once  they  have  taken  the 
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time  to  analyze  the  data  and  understand  how  they  can  best  respond  to  their  customers,  as 
well  as  meet  the  needs  of  potential  customers  to  win  their  loyalty  and  business  (Jones, 
1998).  The  private  sector  has  embraced  the  concept  of  knowledge-based  decision¬ 
making  because  it  provides  a  means  of  having  a  strong  foundation  from  which  to  operate 
and  from  which  to  determine  what  changes  are  needed  for  the  future. 

The  public  sector,  however,  seems  to  use  the  concept  of  knowledge-based 
decision-making  in  a  slightly  different  way  as  compared  to  the  public  sector.  Public 
sector  organizations,  from  the  federal  government  to  the  state  and  local  levels,  use 
knowledge-based  decision-making  as  a  way  to  determine  outcomes  from  the  way  in 
which  money  and  other  resources  are  used  in  relation  to  policy  initiatives  (Metcalfe, 
2006).  In  some  respects,  the  use  of  knowledge-based  decision-making  seems  to  occur 
more  often  after  resources  have  been  used  as  opposed  to  before.  For  example,  data  is 
often  collected  with  regards  to  how  additional  spending  for  education  initiatives  have 
impacted  test  scores,  or  how  many  people  are  provided  access  to  health  care  services 
because  of  additional  spending  to  open  free  health  clinics. 

Most  public  sector  organizations,  especially  at  the  federal  level,  are  often  large 
and  complex.  Since  the  process  of  knowledge-based  decision-making  is  largely 
dependent  upon  the  size  and  complexity  of  an  organization,  it  can  be  hard  to  implement 
in  the  public  sector  (Miller  &  Berger,  2003).  This  provides  some  understanding  of  why 
public  sector  organizations  appear  to  use  the  concepts  of  knowledge-based  decision¬ 
making  in  a  different  way  as  compared  to  the  private  sector.  This  is  certainly  true  in 
terms  of  the  federal  government  in  which  actions  and  initiatives  must  be  taken  for 
millions  of  people  in  different  locations  and  with  a  variety  of  different  needs.  However, 


3 


once  a  decision  is  made,  then  data  can  be  collected  about  the  outcomes  resulting  from  the 
resources  that  were  used.  This  knowledge  is  used  to  determine  if  the  return  on 
investment  with  regard  to  the  resources  that  were  expended  is  favorable,  which  leads  to 
better  decision-making  in  the  future. 

Regardless  of  the  sector  in  which  it  is  used,  the  underlying  principle  for 
knowledge-based  decision-making  is  the  transfer  of  knowledge  to  a  group  of  people  or  an 
organization  to  increase  the  knowledge  that  exists  about  a  specific  situation,  clients,  or 
work  processes  (Srikantaiah  &  Koenig,  2008).  Additionally,  the  result  of  this  process  is 
that  knowledge  has  been  gained  by  the  organization  in  question  and  decisions  are  based 
on  actual  information  and  data  instead  of  personal  opinions  or  perceptions.  The  ultimate 
benefit  is  the  ability  to  make  decisions  based  on  known  conditions  that  positively  impact 
an  organization  (Salas  &  Maurino,  2010).  This  positive  impact  will  allow  organizations 
to  better  respond  to  the  conditions  that  exist  as  opposed  to  simply  trusting  that  the 
decisions  that  are  made  will  meet  the  needs  of  the  organization. 

Research  Problem 

There  is  a  lack  of  the  adoption  of  knowledge-based  decision-making  within  the 
DoD  acquisition  process.  The  acquisition  process  takes  into  consideration  many  forms  of 
information  products  regarding  programs  both  by  federal  statute  and  DoD  regulation. 
However,  the  milestone  decisions  based  on  these  information  requirements  have  not 
translated  into  better  program  outcomes.  Additionally,  milestone  decisions  are  normally 
made  when  all  statutory  and  regulatory  information  requirements  are  satisfied  and  not 
based  on  when  critical  technology,  design,  and  manufacturing  knowledge  is  attained.  As 
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a  result,  DoD  acquisition  programs  continue  to  experience  problems  with  both  increased 
cost  and  schedule  and  the  degradation  of  performance  and  quality. 

Research  Objective 

To  address  this  problem,  DoD  acquisition  officials  require  improved  decision-making 
information  at  key  program  junctures  that  capture  all  program  goals,  objectives,  and 
concerns  to  facilitate  better  knowledge-based  decisions.  Therefore,  the  objective  of  this 
research  is  to  assess  the  use  of  knowledge-based  decision  support  in  the  DoD  acquisition 
process.  To  address  this  objective,  the  research  attempted  to  answer  the  following 
investigative  questions. 

1 .  What  are  the  critical  pieces  of  information  that  are  required  for  an  informed 
decision  at  key  decision  points  in  the  acquisition  process? 

2.  How  does  the  currently  available  information  (contained  in  required  decision 
products)  compare  to  what  is  needed  for  decisions  with  regard  to  technology, 
design,  and  production  knowledge  for  each  of  the  key  decision  points  in  the 
acquisition  process? 

3.  What  is  the  effect  of  this  lack  of  information  on  DoD  acquisition  programs? 

4.  Can  this  effect  be  quantified  in  terms  of  cost,  schedule,  and  performance? 

Methodology 

To  help  answer  these  investigative  questions,  a  secondary  data  source  was 
examined  and  compared  with  the  knowledge  required  by  either  federal  statute  or  DoD 
regulation  at  key  decision  points.  Both  entities  were  also  compared  with  knowledge- 
based  decision-making  best  practices  identified  through  the  literature  review.  The  data 
consisted  of  responses  from  senior-level  Air  Force  acquisition  officials  interviewed  by 
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the  Air  Force  Acquisition  Chief  Process  Office  regarding  the  acquisition  decision-making 
process.  These  interviews  helped  identify  data  products  used  during  decision-making  and 
how  those  products  influenced  decisions  at  key  milestones.  After  summarizing  the 
interview  data  using  descriptive  statistics,  data  reduction  was  accomplished  using 
Microsoft  Excel  software  to  perform  content  analysis  and  identify  themes  represented 
within  the  interview  data.  This  facilitated  the  comparison  of  the  data  with  required 
knowledge  and  best  practices  used  with  knowledge-based  decision-making  during  the 
acquisition  process. 

Thesis  Outline 

In  the  chapters  that  follow,  the  findings  of  the  in-depth  literature  review,  research 
methodology  and  analysis,  and  results  will  be  presented.  The  study  will  establish  if  the 
intention  and  usefulness  of  required  information  at  key  decision  points  in  acquisition 
programs  provides  sufficient  knowledge  to  facilitate  more  informed  decisions.  It  will 
examine  if  key  decisions  are  made  at  the  appropriate  time  within  the  defense  acquisition 
process.  The  literature  review  in  Chapter  II  will  provide  the  context  of  the  DoD 
acquisition  process,  the  information  required  by  federal  statute  and  DoD  regulation,  and 
best  practices  with  regard  to  knowledge-based  decision-making.  Additionally,  it  will 
provide  a  comparison  of  knowledge-based  decision-making  in  the  private  and  public 
sector.  In  Chapter  III,  the  methodology  and  research  design  will  be  discussed;  this  is 
followed  by  an  analysis  of  the  data  in  Chapter  IV.  The  final  chapter  will  answer  the 
research  questions  and  provide  recommendations  for  improving  knowledge-based 
decision-making  in  DoD  acquisition  programs. 
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Chapter  II.  Literature  Review 


To  focus  this  research  effort  on  the  knowledge  required  to  make  decisions  for 
acquisition  programs  at  key  decision  points  in  the  Department  of  Defense  (DoD) 
acquisition  process,  the  literature  review  is  divided  into  three  parts.  The  first  is  an 
examination  of  the  differences  in  the  acquisition  practices  in  the  public  and  private  sector. 
The  second  offers  a  baseline  for  the  statutory  and  regulatory  knowledge  requirements  of 
the  DoD  acquisition  system  at  key  milestones.  Finally,  the  third  section  reviews  studies 
regarding  best  practices  of  knowledge-based  decision  support. 

Acquisition  Practices:  Military  versus  Private  Sector 

There  are  some  notable  differences  in  terms  of  how  acquisition  efforts  work 
within  the  DoD  and  similar  efforts  in  the  private  sector.  Some  of  these  variations  are 
primarily  due  to  the  differences  in  organizational  goals.  Ferguson  and  DeRiso  (1994) 
completed  a  detailed  study  of  software  acquisition  efforts  between  the  DoD  and  the 
private  sector  in  terms  of  defining  program  requirements,  selecting  a  contractor  (i.e., 
source  selection),  test  and  evaluation,  and  the  development  process.  Although  their 
research  dealt  primarily  with  software  acquisitions,  the  findings  are  applicable  to  the 
acquisition  process  of  traditional  weapon  systems. 

Defining  Program  Requirements 

In  terms  of  defining  the  requirements  for  a  system,  the  DoD  generates 
requirements  based  on  the  needs  of  an  operational  user.  This  requirements  generation 
process,  called  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS),  is 
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focused  on  identifying  requirements  for  weapon  systems  for  the  entire  DoD  as  opposed  to 
the  individual  services  (CJCSI  3170.01G,  2009).  However,  in  a  2008  study,  the  General 
Accounting  Office  (GAO)  found  that  the  process  has  yet  to  be  effective  in  this  manner. 
According  to  the  report,  nearly  70  percent  of  the  requirements  were  sponsored  by  the 
individual  services  as  opposed  to  the  joint  community  (GAO,  2008).  Additionally,  it 
noted  that  nearly  almost  all  of  these  requirements  were  approved.  As  a  result,  there  has 
not  been  any  notable  fiscal  efficiency  gained  through  the  JCIDS  process.  In  fact,  the 
remaining  costs  of  major  weapon  systems  have  increased  significantly  since  JCIDS  was 
implemented.  Figure  1  depicts  the  cost  remaining  versus  the  annual  investment 
appropriations. 


Major  Defense  Acquisition  Program  Costs  Remaining  versus  Annual  Appropriations,  from 
Fiscal  Year  2000  through  Fiscal  Year  2007 
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Figure  1.  Cost  Remaining  versus  Annual  Investment  Appropriations  (GAO,  2008) 


A  critical  factor  within  the  requirements  generation  process  is  the  time  it  takes  to 
validate  a  requirement.  In  the  aforementioned  report,  the  GAO  (2008)  found  that  the 
average  time  to  validate  a  DoD  requirement  was  10  months.  In  the  private  sector  though, 
it  is  not  unusual  for  companies  to  validate  requirements  on  a  quarterly  basis  (Bate  and 


Roberts,  2002).  The  responsiveness  associated  with  validating  a  requirement  can  mean 
the  ability  to  respond  to  a  threat  or  beating  a  competitor  to  market  with  a  new  product 
(Dasarathy,  1985). 

Accurately  capturing  key  system  requirements  early  in  the  acquisition  process  has 
a  direct  impact  on  the  outcome  of  a  program.  However,  a  preponderance  of  DoD 
acquisition  programs  fail  to  do  so.  As  a  result,  most  of  these  programs  changed  key 
system  requirements  after  the  start  of  development.  These  changes  translated  into 
increased  costs  and  schedule  delays  (GAO,  2010).  In  an  attempt  to  reduce  changes  in 
requirements,  the  DoD  issued  a  change  in  policy  requiring  programs  to  hold 
configuration  steering  boards  to  make  certain  that  the  effect  of  cost  and  performance  are 
considered  when  making  major  technical  adjustments  (GAO,  2010). 

Source  Selections 

Another  key  difference  between  the  DoD  and  private  sector  is  the  communication 
with  vendors  prior  to  source  selection.  Within  the  DoD,  the  involvement  of  vendors  and 
contractors  is  often  avoided  until  a  specific  vendor  or  contractor  has  been  chosen,  thus 
reducing  the  sharing  of  information  and  knowledge  that  is  used  in  the  decision-making 
process.  In  the  private  sector,  vendors  and  contractors  are  encouraged  to  become 
involved  very  early  in  the  process  to  leverage  their  knowledge  and  information  (Ferguson 
andDeRiso,  1994). 

In  terms  of  vendor  or  contract  selection,  the  DoD  typically  operates  by  requiring 
vendors  or  contractors  to  compete  against  each  other,  rather  than  collaborate  to  combine 
knowledge  sets  and  abilities  to  win  a  contract  as  a  unified  team.  The  selection  of  a 
vendor  or  contactor  also  relies  on  a  very  specific  set  of  standards  that  cannot  be  changed 
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or  deviated  from  in  any  way.  If  any  of  the  source  selection  metrics  created  in  advance  is 
not  met  by  a  particular  vendor  or  contractor,  the  vendor  or  contractor  is  often  removed 
from  the  selection  process  without  any  consideration  regarding  whether  their  knowledge 
set  might  be  useful  for  the  project.  Again,  this  is  different  from  acquisition  efforts  in  the 
private  sector  in  which  vendors  are  typically  allowed  to  work  together  in  order  to  obtain 
the  highest  level  of  knowledge  and  ability  from  those  that  can  best  serve  the  project. 
Private  sector  vendor  or  contractor  selection  also  has  a  selection  process  that  is  much  less 
stringent  and  allows  for  more  flexibility  based  on  the  recommendations  and  information 
obtained  by  the  source  selection  team  (Ferguson  and  DeRiso,  1994).  Table  1  shows  the 
differences  in  vendor  selection  in  the  private  sector  and  the  DoD. 


Table  1.  Best  Practice  -  Vendor  Selection  (Ferguson  and  DeRiso,  1994) 


Vendor  Selection 

Best  Commercial  Practice 

C li nrent  DoD  Practice 

Go hch.  multiple  {but  not  ell)  qualified  vendors. 
Encourage  teaming  with  a  view  to  attaining  a 
relationsfii  p  that  covers  the  entire  life  cycle  and 
fosters  tradeoffs  in  cost  and  schedule. 

Solicit  all  possible  vendors.  Vendor  proposals 
must  meet  t G-lI'W:.  of  requirements.  Teaming 
seldom  encouraged.;  development  and 
maintenance  usually  separate  entities. 

Compare  vend  or  history  and  experienc  e.  Maintain 
long-term  relationships. 

Can  compare  previous  performance,  but  normally 
can't  have  long-term  relationship*. 

The  organization  that  will  be  responsible  for  a 
system  over  its.  full  life  cycle  is  heavily  involved 
front  the  beginning. 

Maintenance  organization  nest  usually  involved  in 
vendor  selection  process. 

Use  site  visits  and  demonstrations  to  gain 

Knowledge  of  vendor  cap  a  b-iiisies. 

Site  visil.  only  by  capability  evaluation  team,  or 
ash-er  expert  teams.  Visits  are  very  s-tru-ctune-d. 

Overall  goals:  (1}  obtain  product  at  reasonable 
cost  as  soon  as  possible:  and  (2)  achieve  the 
business  case  for  the  system. 

Overall  goal:  Obtain  lowest  cost  prod uc&  that 
rigorously  meets  all  requirements,  bui  be-  fair. 

Relatively  few  review  and  approval  steps  once 
vendor  is  selected. 

Review  and  approval  process  mare  structured 
and  complex  once  vendor  selected. 

Fast  performance  weighted  heavily  (sometimes 
primary  factor)  in  selection  process. 

Fast  performance  considered,  but  usually  only  ss 
a  mi  nor  factor. 

More  flexibility  in  vendor  selection  based  on 
metrics  and  overall  assessment. 

Selection  of  vendor  forced  by  use  of  predefined 
metrics  for  proposal  evaluation. 

S  u  m  m  a  r  v 

Very  different  processes  with  commercial  much  more  flexible  but  with  no  requirement  for  fairness,  or  to  maintain 
the  public  trust.  Commercial  encourages  vendors  to  offer  best  solution,  but  solution  may  not  meet  f  al  the 

requirements.  T earning  and  long-term  relationships  are  more  easily  a  oco  it,  modeled  by  industry. 

Test  and  Evaluation 
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Test  and  evaluation  is  the  next  key  knowledge  area  explored  in  this  research. 

Like  the  DoD,  leading  private  sector  firms  have  dedicated  test  organizations  for  new 
products.  However,  in  general,  these  organizations  are  aligned  under  the  program 
manager  and  testing  is  often  performed  in  a  single  facility  by  the  same  team  members 
(Ferguson  and  DeRiso,  1994).  The  DoD  conversely  uses  independent  organizations  to 
test  systems  (DoD  Test  and  Evaluation  Guide,  2005).  As  such,  there  is  opportunity  for 
gaps  in  knowledge  transfer.  Additionally,  the  GAO  (20 1 0)  found  that  programs  are  not 
testing  prototypes  that  are  production  representative  before  committing  to  production.  In 
doing  so,  the  maximum  benefit  of  test  and  evaluation  is  not  realized  due  to  the  possibility 
of  changes  to  the  design  after  production.  Figure  2  shows  the  GAO  findings  with  regard 
to  the  number  of  programs  testing  a  production  representative  prototype  before  and  after 
a  production  decision. 

Perhaps  the  biggest  difference  in  test  and  evaluation  is  that  the  private  sector 
makes  less  of  a  distinction  between  developmental  and  operational  testing  (Ferguson  and 
DeRiso,  1994).  The  reason  is  two-fold:  first,  by  the  time  the  private  sector  makes  a 
decision  to  field  a  new  product,  the  technology  is  proven.  Second,  the  cycle  time  in 
product  development  is  typically  much  longer  in  the  DoD.  Therefore,  more  technical  risk 
is  assumed  early  in  the  life-cycle  while  anticipating  that  the  technology  readiness  level 
will  mature. 
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Figure  2.  Testing  of  Production-Ready  Prototypes  (GAO,  2008) 
Development  Process  Requirements 

Another  notable  difference  between  the  acquisition  process  within  the  DoD  and 
the  private  sector  can  be  attributed  to  the  rigidity  of  the  process  for  product  development. 
Strict  requirements  imposed  by  Congress  and  the  DoD  are  inflexible  and  largely  based  on 
a  single  project,  rather  than  understanding  existing  systems  and  the  impact  of  future 
projects  that  may  be  implemented.  In  essence,  a  large  part  of  the  acquisition  process  is 
focused  on  following  the  process  itself  and  not  on  how  knowledge  can  be  applied  to 
making  more  informed  decisions  (Defense  Acquisition  Performance  Assessment,  2006). 
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Knowledge  Requirements  of  the  Defense  Acquisition  Process 

The  DoD  acquisition  process  exists  to  provide  a  secure  and  supportable  military 
to  maintain  our  national  security  strategies  (Defense  Acquisition  Guidebook,  2009).  The 
process  is  governed  by  the  Federal  Acquisitions  Regulation  (FAR)  and  implemented  by 
DoD  Instruction  (DoDI)  5000.02.  The  DoDI  5000.02  provides  the  framework  in  which  a 
program  progresses  throughout  its  life-cycle.  Figure  3  illustrates  the  life-cycle 
framework  view  of  the  acquisition  process.  In  addition  to  the  decisions  at  Milestones  A, 
B,  and  C  (which  authorize  entrance  into  the  major  program  phases),  there  are  three  other 
key  decision  points  within  this  framework  that  must  be  made  by  the  Milestone  Decision 
Authority  (MDA)  (Young,  Grimes,  and  McQueary,  2008).  These  decision  points  (also 
noted  in  Figure  3)  are  Materiel  Development  Decision,  Post-Critical  Design  Review,  and 
Full  Rate  Production  Decision  Review. 


Lifecycle  Framework  View 
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Figure  3.  Decision  Points  in  DoD  Acquisition  Process  (DoDI  5000.02,  2009) 
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The  acquisition  process  itself  is  initiated  by  a  validated  need  identified  by  the 
military  user.  The  need  can  stem  from  a  new  threat,  capability  gap,  or  even  a  new 
technological  opportunity.  For  this  need  to  be  addressed  via  the  acquisition  process,  a 
determination  is  made  whether  a  Materiel  or  Non-Materiel  Solution  is  viable.  This 
decision  is  made  at  the  Materiel  Development  Decision  (MDD)  review,  which  is  the 
formal  entry  point  into  the  acquisition  process  and  is  mandatory  for  all  programs.  At  this 
decision  point,  there  is  not  any  statutory  knowledge  requirement.  However,  the 
knowledge  required  by  regulation  includes  an  Analysis  of  Alternatives  and  an  Initial 
Capabilities  Document. 

If  a  Materiel  Solution  is  deemed  the  better  alternative,  the  program  is  authorized 
to  enter  the  Materiel  Solution  Analysis  (MSA)  phase.  The  purpose  of  the  MSA  phase  is 
to  review  potential  materiel  solutions  and  satisfy  the  entrance  criteria  for  the  next 
program  milestone  designated  by  the  MDA.  The  Analysis  of  Alternatives  (AoA)  Study 
Guidance  and  the  Initial  Capabilities  Document  (ICD)  provide  direction  for  the  activities 
in  this  phase.  The  AoA  focuses  on  identifying  and  analyzing  alternatives,  measures  of 
effectiveness,  cost,  schedule,  concepts  of  operations,  and  overall  risk  (Young,  Grimes, 
and  McQueary,  2008).  The  ICD  provides  the  preliminary  concept  of  operations,  a 
description  of  the  capability,  the  operational  risk,  and  the  basis  for  determining  that  non¬ 
materiel  alternatives  will  not  adequately  alleviate  the  capability  gap.  The  MSA  phase 
ends  when  the  AoA  has  been  completed,  materiel  solution  options  for  the  capability  need 
identified  in  the  approved  ICD  have  been  recommended,  and  the  phase-specific  entrance 
criteria  for  the  initial  review  milestone  have  been  satisfied  (Young,  Grimes,  and 
McQueary,  2008). 
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The  next  decision  point  is  the  Milestone  A  review.  This  decision  authorizes  entry 
into  the  Technology  Development  phase.  The  purpose  of  this  phase  is  to  reduce 
technology  risk,  determine  and  mature  the  proper  set  of  technologies  to  be  included  in  the 
system,  and  demonstrate  that  the  technology  performs  on  prototypes.  Table  2  depicts  the 
key  statutory  and  regulatory  information  required  of  Major  Defense  Acquisition 
Programs  (MDAP)  for  this  phase.  The  complete  set  of  information  requirements  for 
Milestone  A  can  be  found  in  Appendix  A. 


Table  2.  Key  Milestone  A  Statutory/Regulatory  Information 


STATUTORY  INFORMATION 
REQUIRED 

APPLICABLE  STATUTE,  POLICY,  AND 
OTHER  REFERENCES 

Analysis  of  Alternatives 

Subtitle  III  of  title  40,  U.S.C.,  Section  2366a  of  Title 

10 

Consideration  of  Technology  Issues 

Paragraph  (b)(5)  of  Section  2364  of  Title  10,  U.S.C. 

Independent  Cost  Estimate 

Section  2434  of  Title  10,  U.S.C. 

Milestone  Decision  Authority  Program 
Certification 

Section  2366a  of  Title  10,  U.S.C. 

Technology  Development  Strategy  (TDS) 

Section  803  of  Public  Law  107-314 

REGULATORY  INFORMATION 
REQUIRED 

APPLICABLE  REGULATION,  POLICY,  AND 
OTHER  REFERENCES 

Initial  Capabilities  Document 

CJCS  Instruction  3170.01 

Life-Cycle  Support  Plan 

DoD  Directive  5250.01 

Systems  Engineering  Plan 

DoD  Instruction  5000.02 

Test  and  Evaluation  Strategy 

DoD  Instruction  5000.02 

The  next  key  decision  point  is  the  Milestone  B  review.  This  decision  authorizes  a 
program  to  enter  the  Engineering,  Manufacturing,  and  Development  (EMD)  phase.  The 
purpose  of  this  phase  is  to  develop  a  weapon  system  or  an  increment  of  capability.  The 
entrance  criteria  for  this  phase  depend  heavily  upon  technology  maturity,  mature/stable 
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requirements,  and  full  funding.  Entrance  into  EMD  also  signals  the  beginning  of  a 
formal  acquisition  program.  Table  3  shows  both  the  key  statutory  and  regulatory 
information  required  for  this  decision  (Young,  Grimes,  and  McQueary,  2008).  The  full 
set  of  requirements  can  be  found  in  Appendix  B. 


Table  3.  Key  Milestone  B  Statutory/Regulatory  Information 


STATUTORY  INFORMATION 

APPLICABLE  STATUTE,  POLICY,  AND  OTHER 

REQUIRED 

REFERENCES 

Acquisition  Program  Baseline 

Section  2435  of  title  10,  U.S.C. 

Consideration  of  Technology  Issues 

Paragraph  (b)(5)  of  Section  2364  of  Title  10,  U.S.C. 

Independent  Cost  Estimate 

Section  2434  of  Title  10,  U.S.C. 

Low-Rate  Initial  Production  Quantities 

Section  2400  of  Title  10,  U.S.C. 

Live  Fire  Test  and  Evaluation  Plan 

Section  2366  of  Title  10,  U.S.C. 

REGULATORY  INFORMATION 

APPLICABLE  REGULATION,  POLICY,  AND 

REQUIRED 

OTHER  REFERENCES 

Acquisition  Strategy 

DoD  Instruction  5000.02 

Technology  Readiness  Assessment 

DoD  Instruction  5000.02 

Capabilities  Development  Document 

CJCS  Instruction  3170.01 

Life-Cycle  Support  Plan 

DoD  Directive  5250.01 

Systems  Engineering  Plan 

DoD  Instruction  5000.02 

Test  and  Evaluation  Master  Plan 

DoD  Instruction  5000.02 

The  EMD  phase  contains  two  major  elements:  Integrated  System  Design  (ISD) 
and  System  Capability  and  Manufacturing  Process  Demonstration  (SD&MPD).  The  ISD 
identifies  system  and  system-of-systems  functionality  and  interfaces,  completes  design 
for  hardware  and  software,  and  reduces  system-level  risk.  The  SD&MPD  reveals  the 
capability  of  the  system  to  function  as  designed  and  be  manufactured.  In  addition  to 
these  two  efforts,  the  EMD  phase  also  includes  a  major  decision  point:  the  Post-Critical 
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Design  Review  (P-CDR)  Assessment.  This  assessment  offers  an  opportunity  to  review 
design  maturity  (Young,  Grimes,  and  McQueary,  2008). 

The  next  and  final  key  decision  point  addressed  in  this  research  is  Milestone  C. 
This  milestone  authorizes  entrance  into  the  Production  and  Deployment  phase.  The 
intent  of  this  phase  is  to  demonstrate  that  the  production  system  can  operate  in 
accordance  with  the  user’s  requirements.  Additionally,  this  phase  demonstrates  that  the 
production  system  can  be  manufactured  (Young,  Grimes,  and  McQueary,  2008).  Table  4 
shows  the  key  statutory  and  regulatory  information  requirements  for  Milestone  C.  The 
full  set  of  requirements  can  be  found  in  Appendix  C. 


Table  4.  Key  Milestone  C  Statutory/Regulatory  Information  Requirements 


STATUTORY  INFORMATION 

APPLICABLE  STATUTE,  POLICY,  AND  OTHER 

REQUIRED 

REFERENCES 

Acquisition  Program  Baseline 

Section  2435  of  title  10,  U.S.C. 

Consideration  of  Technology  Issues 

Paragraph  (b)(5)  of  Section  2364  of  Title  10,  U.S.C. 

Independent  Cost  Estimate 

Section  2434  of  Title  10,  U.S.C. 

Manpower  Estimate 

Section  2434  of  Title  10,  U.S.C. 

Analysis  of  Alternatives 

Subtitle  III  of  title  40,  U.S.C.,  Section  2366a  of  title  10 

REGULATORY  INFORMATION 

APPLICABLE  REGULATION,  POLICY,  AND 

REQUIRED 

OTHER  REFERENCES 

Acquisition  Strategy 

DoD  Instruction  5000.02 

Technology  Readiness  Assessment 

DoD  Instruction  5000.02 

Capabilities  Production  Document 

CJCS  Instruction  3170.01 

Life-Cycle  Support  Plan 

DoD  Directive  5250.01 

Systems  Engineering  Plan 

DoD  Instruction  5000.02 

Test  and  Evaluation  Master  Plan 

DoD  Instruction  5000.02 

Independent  Technology  Readiness 
Assessment  (if  required) 

DoD  Instruction  5000.02 
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Knowledge-Based  Decision  Support  Best  Practices 

The  GAO  has  completed  extensive  research  with  regard  to  knowledge-based 
decision-making  in  major  system  acquisitions.  In  fact,  the  GAO  (2008)  published  their 
findings  regarding  best  practices  for  developing  new  products  and  stated  the  following: 


Good  acquisition  outcomes  require  the  use  of  a  knowledge-based  approach  to 
product  development  that  demonstrates  high  levels  of  knowledge  before 
significant  commitments  are  made.  Achieving  the  right  knowledge  at  the  right 
time  enables  leadership  to  make  informed  decisions  about  when  and  how  best  to 
move  into  various  acquisition  phases.  In  essence,  knowledge  supplants  risk  over 
time.  This  building  of  knowledge  consists  of  information  that  should  be  gathered 
at  three  critical  points  over  the  course  of  a  program,  (p.5) 

In  a  2010  study,  the  GAO  assessed  the  knowledge  attained  on  42  major  defense 

acquisition  programs  (MDAP)  at  key  milestones  early  in  the  program’s  life-cycle  (prior 

to  Milestone  C).  To  support  the  study,  the  GAO  collected  data  on  programs  with  regard 

to  technology  maturity,  design  stability,  and  production  maturity  from  August  2009  to 

March  2010.  The  study  centers  on  the  following  three  key  junctures  for  knowledge 

points. 


-  Knowledge  Point  1 :  Requirements  and  technological  capability  are  matched 

-  Knowledge  Point  2:  Knowledge  that  design  will  work  as  required 

-  Knowledge  Point  3 :  Knowledge  that  the  design  can  be  produced  within  cost, 
schedule,  and  quality  targets 

Figure  4  shows  these  knowledge  points  in  relation  to  key  phases  in  the  product 
development  cycle. 
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Figure  4.  Best  Practices  in  Product  Development  (GAO,  2008) 


The  first  of  these  knowledge  points  suggests  that  a  match  should  occur  between 
what  is  needed  and  what  is  available  in  terms  of  technology,  design,  and  funding.  This 
knowledge  point  occurs  after  the  completion  of  the  technology  development  phase.  A 
good  gauge  of  whether  a  match  is  made  is  noting  the  level  of  technology  maturity  at  the 
beginning  of  the  development  stage.  A  match  occurs  when  a  program  has  demonstrated 
that  the  critical  technologies  have  been  verified  to  work  in  their  intended  environment 
(GAO,  2008).  The  GAO  (2010)  found  that  programs  in  general  are  not  conducting 
systems  engineering  reviews  early  in  the  program  to  ensure  a  match  between  resources 
and  requirements. 

The  second  knowledge  point  represents  the  fact  that  a  product’s  design  should  be 
demonstrated  to  function  as  planned  and  meet  the  requirements  that  have  been 
established  (GAO,  2008).  For  DoD  milestone  decision  authorities,  this  means  being 
assured  that  the  system  design  is  stable  and  will  perform  in  a  way  that  was  expected  by 
the  user.  According  to  the  GAO,  program  stability  should  be  reached  by  the  halfway 
point  of  system  development.  An  indicator  of  design  stability  is  the  completion  of  at 
least  90  percent  of  engineering  drawings  by  the  Critical  Design  Review  (GAO,  2008). 
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Additionally,  the  engineering  drawings  and  designs  should  accurately  reflect  the  results 
obtained  from  testing  the  system  in  question. 

This  leads  to  the  third  knowledge  point,  which  suggests  that  a  product  should  only 
be  considered  reliable  when  it  can  be  created  within  the  stated  costs,  schedule,  and 
quality  levels.  An  important  indication  of  whether  a  product  or  system  is  reliable  is  when 
it  can  be  created  over  and  over  with  the  same  level  of  performance  and  reliability  (Garrett 
and  Rendon,  2005).  To  facilitate  reliability,  a  best  practice  is  to  make  certain  that  all 
critical  manufacturing  processes  are  in  statistical  control  at  the  start  of  production  (GAO, 
2008).  A  program  should  ensure  that  all  critical  manufacturing  knowledge  is  attained 
before  entering  production.  If  achieved,  then  the  program  will  have  a  stable 
manufacturing  process  that  will  work  as  intended  and  meet  cost,  schedule,  and  quality 
objectives.  Figure  5  is  an  example  of  how  the  best  practices  can  be  used  to  assess  an 
acquisition  program.  The  desired  level  (dotted  line)  is  the  indicator  of  where  the  program 
should  be  in  terms  of  best  practice.  The  hypothesis  is  the  closer  the  program  is  to  the 
desired  level;  the  more  likely  the  program  will  be  within  cost  and  schedule  (GAO,  2010). 
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Figure  5.  Product  Knowledge  as  Compared  with  Best  Practices  (GAO,  2008) 

Application  of  GAO  Knowledge  Points:  National  Aeronautics  and  Space  Administration 
To  apply  the  knowledge  points,  the  GAO  (2005)  reviewed  and  analyzed  the 
National  Aeronautics  and  Space  Administration  (NASA)  project  management  policies 
and  compared  them  to  the  GAO’s  best  practices  on  knowledge-based  decision  making. 
The  study  was  primarily  focused  on  the  Goddard  Space  Flight  Center,  the  Jet  Propulsion 
Lab,  Johnson  Space  Center,  and  Marshall  Space  Flight  Center.  During  its  investigation, 
the  GAO  found  NASA  deficient  in  key  criteria  and  decision  reviews  to  fully  implement  a 
knowledge-based  acquisition  framework. 

There  were,  however,  some  best  practices  followed.  For  example,  NASA 
required  projects  to  hold  a  major  decision  review  before  progressing  from  formulation  to 
implementation.  Additionally,  projects  were  required  to  validate  requirements,  develop 
cost  and  schedule  estimates,  create  preliminary  design,  and  have  an  approved  technology 
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plan.  All  of  these  requirements  were  in  sync  with  the  GAO’s  Knowledge  Point  1 : 
Matching  requirements  with  technology  capability.  The  issue  is  that  NASA  does  not 
require  the  technology  used  in  its  projects  to  be  at  a  high  level  of  maturity  at  that  point. 
The  GAO  insists  that  this  increases  the  risk  that  requirements  will  not  be  met. 

Another  area  found  to  be  deficient  was  the  NASA  policy  of  not  specifying  the 
type  of  reviews  that  project  managers  should  hold  at  key  points  in  the  product 
development  cycle.  The  GAO  indicated  that  technical  reviews  allow  decision-makers  to 
acquire  key  knowledge  at  critical  junctures  in  the  project’s  development.  As  a  result,  the 
GAO  concluded  that  NASA  was  increasing  the  risk  of  cost  and  schedule  overruns. 

The  last  issue  GAO  found  during  its  study  of  NASA  was  the  non-standard  use  of 
criteria  for  indicators  of  program  success  at  key  decision  points.  This  resulted  in  each 
center  within  NASA  reporting  on  different  types  of  project  knowledge  at  key  decision 
points.  At  the  time  of  the  report,  NASA  had  experienced  a  loss  of  experienced  project 
managers  and  system  engineers.  This  loss  of  key  personnel  combined  with  lack  of 
standardized  criteria  exacerbated  the  problem  (GAO,  2005). 

The  GAO  (2005)  made  several  recommendations  to  improve  NASA  management 
policies.  Two  recommendations  centered  on  NASA  using  a  knowledge-based  acquisition 
approach  to  improve  its  decision-making.  As  such,  it  recommended  that  NASA  take  the 
following  actions: 

1 .  Require  the  capture  of  specific  knowledge  to  be  used  as  criteria  for  allowing 
projects  to  enter  implementation  and  proceed  through  development  and  to  support 
informed  investment  decisions. 

2.  Institute  additional  reviews  for  projects  during  project  implementation,  which 
result  in  recommendations  to  the  appropriate  decision  authority. 
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In  response  to  the  GAO  findings,  NASA  issued  a  revised  acquisition  policy  in 
2007.  The  new  policy  instituted  Key  Decision  Points  (KDP)  in  the  development  life- 
cycle  for  NASA’s  space  flight  programs  and  ground  station  projects.  Additionally,  the 
policy  established  a  decision  authority  at  each  KDP  responsible  for  authorizing  the 
entrance  into  the  next  phase.  The  policy  also  required  new  technologies  to  be  sufficiently 
mature  at  the  preliminary  design  review  (GAO,  2009). 

In  a  follow-up  study,  the  GAO  (2009)  again  reviewed  NASA’s  projects.  In  this 
review,  the  GAO  assessed  cost  and  schedule  data  for  13  programs.  They  found  that  10  of 
the  13  programs  experienced  an  average  cost  growth  of  13  percent  (based  on  2008  data) 
since  the  GAO  review  in  2005.  The  average  schedule  delay  for  the  13  programs  was  1 1 
months.  GAO  concluded  that  in  spite  of  the  new  policy,  the  lack  of  knowledge  at  key 
junctures  and  the  continued  use  of  immature  technologies  continued  to  contribute  to  cost 
growth  and  schedule  delay. 

Application  of  Statistical  Decision  Model:  University  of  Southern  California 

The  study  of  the  use  of  knowledge-based  decision  methodologies  in  DoD 

acquisitions,  such  as  the  use  of  statistical  tests  and  models,  is  an  area  that  has  been  largely 

unexplored.  A  study  by  Cohen,  Rolph,  and  Steffey  (1988)  examined  the  statistical 

techniques  in  the  design  and  evaluation  of  operational  test  and  evaluation.  One  of  the 

conclusions  from  that  study  notes  the  following: 

For  many  defense  systems,  the  current  operational  testing  paradigm  restricts  the 
application  of  statistical  techniques  and  thereby  reduces  their  potential  benefits  by 
preventing  the  integration  of  all  available  and  relevant  information  for  use  in 
planning  and  carrying  out  tests  and  in  making  production  decisions.  This 
paradigm  is  was  noted  as  the  major  players  in  the  acquisition  process  -  the 
program  manager,  test  organization,  the  contractor,  user,  and  Congress  -  have 
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very  different  (and  sometimes  competing)  perspectives  on  how  knowledge  is 
applied.  As  a  result,  even  in  those  situations  where  knowledge  is  available  that 
could  be  used  to  make  good  programmatic  decisions;  those  that  are  involved  in 
the  decision-making  process  may  be  pressured  to  do  otherwise,  (p.47) 

Boehm,  Port,  Huang,  and  Brown  (2002)  created  a  spatial  model  that  can  be  used  to 
generate  several  acquisition  models  related  to  user  satisfaction,  cost,  and  quality 
constraints.  The  models  take  into  account  the  expectations  of  key  stakeholders,  system 
requirements,  system  features,  and  development  procedures.  The  models  require  a  great 
deal  of  knowledge  from  all  stakeholders  involved  in  the  development  process.  In  their 
study,  this  model  was  used  on  26  University  of  Southern  California  projects.  The  results 
showed  that  24  out  of  26  projects  were  successful  in  terms  of  meeting  cost,  schedule,  and 
performance  objectives.  The  importance  of  these  results  is  that  they  show  that 
knowledge-based  decision  support  models  could  possibly  be  used  effectively  for  DoD 
acquisitions. 

Application  of  RAND ’s  Decision  Framework:  National  Security  Agency 

In  2006,  the  RAND  Corporation  partnered  with  the  Intelligence  Support  Systems 
(ISS)  division  at  the  National  Security  Administration  to  pilot  RAND’s  Portfolio 
Management  (PortMan)  Decision  Framework.  The  framework  can  be  used  for  Research 
and  Development  (R&D)  projects  or  Operations  and  Maintenance  (O&M)  projects.  The 
purpose  of  the  model  is  to  assess  the  Expected  Value  of  a  group  of  actual  or  proposed 
projects. 

The  Expected  Value  can  then  be  used  as  knowledge  upon  which  to  make 
decisions  that  maximize  the  value  of  R&D  funding  for  a  portfolio.  RAND’s  PortMan 
model  computes  the  Expected  Value  of  a  R&D  project  from  two  primary  knowledge 
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factors:  value  of  successful  implementation  and  the  probability  of  successful 
implementation.  The  value  of  successful  implementation  is  based  on  the  value  of  the 
capability  to  the  organization  and  the  extent  to  which  the  performance  potential  matches 
the  resources  required  to  achieve  the  capability.  The  probability  of  successful 
implementation  is  a  measure  of  the  risk  associated  with  implementing  an  R&D  project  or 
sustaining  an  O&M  project  (RAND,  2004).  The  Expected  Value  (EV)  of  a  project  is 
defined  as  the  Value  of  Successful  Implementation  multiplied  by  the  Probability  of 
Successful  Implementation. 

For  this  particular  study,  RAND  (2006)  developed  two  different  sets  of  metrics 
for  estimating  the  Expected  Value.  They  created  one  set  for  R&D  projects  and  another 
set  for  Operations  and  Maintenance  (O&M)  projects.  A  total  of  17  projects  were 
evaluated  in  the  pilot  study.  To  estimate  the  three  factors  for  the  Expected  Value,  RAND 
performed  a  Delphi  exercise  using  the  members  of  the  ISS’s  Senior  Leadership  Group 
(SLG).  The  SLG  were  senior  decision-makers  in  the  ISS  organization.  Each  SLG 
member  was  given  a  series  of  questions  and  a  ranking  scale  to  provide  answers.  After 
several  rounds  to  reach  consensus,  the  answers  were  converted  to  a  numeric  score  and 
averaged  to  provide  values  for  the  Expected  Value  (RAND,  2006). 

The  results  of  the  RAND  study  concluded  that  the  ranking  of  projects  using  the 
decision  model  was  significantly  different  than  the  ranking  from  the  ISS  methodology, 
which  was  based  on  undocumented  metrics.  RAND  (2006)  concluded  the  following:  the 
RAND  PortMan  model  can  be  effectively  applied  to  both  R&D  and  O&M  portfolio 
decisions  and  the  model  can  be  used  for  both  near-term  (single  fiscal  year)  decisions  and 
as  well  as  long-term  decisions. 


25 


Chapter  III.  Methodology 


This  investigation  used  a  qualitative  research  methodology  to  answer  the  research 
questions  offered  in  the  first  chapter.  The  rationale  for  using  qualitative  methods  was 
based  on  the  fact  that  three  different  information  types  were  gathered:  (1)  secondary  data 
consisting  of  survey  responses  to  open-ended  questions,  (2)  knowledge  required  by  either 
federal  statute  or  Department  of  Defense  (DoD)  regulation  at  key  decision  points,  and  (3) 
knowledge-based  decision-making  best  practices.  The  second  and  third  information 
types  were  obtained  through  the  literature  review. 

This  chapter  includes  four  main  sections.  The  first  section  provides  an  overview 
of  the  thesis  research  methodology.  The  second  discusses  the  survey  effort,  while  the 
third  section  describes  how  data  reduction  was  performed  to  analyze  content.  Finally,  the 
fourth  section  summarizes  the  method  used  to  compare  information  types  and  identify 
themes  represented  within  the  data  by  describing  the  assessment  model. 

Overall  Research  Methodology 

The  overall  research  approach  followed  the  nine-step  framework  proposed  by 
Buchanan  (1980)  as  shown  in  Figure  6.  Additionally,  Booth,  Colomb,  and  Williams 
(2003)  suggest  that  an  extensive  research  topic  is  selected  and  then  confined  to  develop  a 
manageable  thesis  statement.  In  that  regard,  this  research  focused  on  “knowledge-based 
decision-making,”  which  was  then  was  confined  to  “knowledge-based  decision-making 
in  DoD  acquisition.”  As  stated  in  Chapter  I,  the  following  investigative  questions  were 
subsequently  established. 
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1 .  What  are  the  critical  pieces  of  information  that  are  required  for  an  informed 
decision  at  key  decision  points  in  the  acquisition  process? 

2.  How  does  the  currently  available  information  (contained  in  required  decision 
products)  compare  to  what  is  needed  for  decisions  with  regard  to  technology, 
design,  and  production  knowledge  for  each  of  the  key  decision  points  in  the 
acquisition  process? 

3.  What  is  the  effect  of  this  resulting  lack  of  information  on  DoD  acquisition 
programs? 

4.  Can  this  effect  be  quantified  in  terms  of  cost,  schedule,  and  performance? 


Step  1 :  Research  Definition  (Chapter  I) 

if 

Step  2\  Literature  Review  (Chapter  n) 

if 

Step  3:  Develop  Research  Constructs  (Chapter  III) 

if 

Step  4:  Design  the  Interview  (Chapter  III) 

l 

Step  5:  Data  Collection  (Chapter  HI) 

I 

Step  6:  Data  Analysis  (Chapter  IV) 

if 

Step  7:  Conclusions  and  Recommendations  (Chapter  V) 


Figure  6.  Research  Methodology  (Buchanan,  1980) 
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This  chapter  thus  focuses  on  steps  five  and  six  of  Buchanan’s  (1980)  framework: 
data  collection  and  data  analysis.  Step  five  (data  collection)  involved  both  secondary 
data  and  extraction  of  data  from  existing  documents.  Step  six  (data  analysis)  consisted  of 
three  parallel  activities  adopted  from  the  analysis  framework  developed  by  Miles  and 
Huberman  (1994).  The  framework  describes  the  major  phases  of  data  analysis  as  data 
reduction,  data  display,  and  conclusion  drawing  and  verification.  Figure  7  illustrates  the 
data  analysis  framework. 


Survey  Effort 

For  this  research  effort,  the  survey  data  was  obtained  from  SAF/ACPO  (Air  Force 
Acquisition  Chief  Process  Office).  SAF/ACPO,  in  conjunction  with  the  Center  for 
Reengineering  and  Enabling  Technology  (CRET),  was  charged  with  reviewing  and 
validating  the  GAO  findings  primarily  with  regard  to  knowledge-based  decision  support 
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and  improving  the  fidelity  of  acquisition  decision-making  within  the  U.S.  Air  Force. 
SAF/ACPO  chose  senior-level  personnel  from  across  the  Air  Force  acquisition 
community.  All  personal  identifiable  information  was  removed  from  the  data  before  it 
was  used  for  this  research. 

Survey  Construction 

The  questions  asked  during  the  survey  are  provided  in  Table  5.  The  questionnaire 
provided  to  the  participants  was  relatively  short.  The  people  solicited  for  this  study  are 
senior  acquisition  officials  with  many  responsibilities.  Providing  them  with  a  long  and 
cumbersome  interview  that  required  a  great  deal  of  time  would  have  likely  resulted  in 
few,  or  possibly  no,  responses.  Furthermore,  with  a  few  very  direct  and  relevant 
questions,  it  can  be  possible  to  get  directly  to  the  core  of  the  issues  being  researched  and 
not  have  to  deal  with  a  great  deal  of  information  and  data  that  may  have  little  or  no 
relevancy  to  the  research  problem  being  addressed. 

The  questions  shown  in  Table  5  were  chosen  for  inclusion  in  the  survey  because 
they  allowed  the  participants  to  provide  information  about  the  type  of  knowledge  that  is 
used  in  the  decision-making  process,  how  other  stakeholders  are  involved  in  the  process, 
and  whether  additional  knowledge  sets  would  be  considered  valuable  in  the  acquisition 
process.  These  questions  provided  a  means  for  not  only  gaining  insight  into  the 
knowledge  set  in  decision-making  process,  but  also  how  that  knowledge  set  is  used  to 
make  decisions. 
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Table  5.  Survey  Questions 


1 .  Does  your  organization  have  any  ongoing  initiatives  that  address  GAO  findings? 
If  so,  what  is  the  initiative  and  what  is  its  purpose? 

2.  What  major  decision  product(s)  does  your  organization  create?  (A  decision  product 
is  defined  as  any  object  (report,  review,  plan,  document  package,  database  update,  etc.) 
that  is  used  to  make  an  Acquisition  Decision.  These  items  include  Budget  Execution 
Documentation,  Justification  Documents,  Monthly  Acquisition  Reports,  etc.  Items  that 
we  are  not  looking  to  identify  are  Weekly  Reports,  Org  Status  Reports,  etc.) 

3.  Who  are  the  customer(s)  of  these  decision  product(s)? 

4.  What  decision  product(s)  do  you  receive? 

5.  What  are  the  source(s)  or  supplier(s)  of  these  products? 

6.  What  feedback  does  your  organization  receive  from  your  customer(s)? 

7.  What  feedback  does/would  your  organization  give  to  your  suppliers(s)? 

8.  What  acquisition  decision(s)  does  your  customer(s)  make  relative  to  the  decision 
product(s)  your  organization  provides  to  them?  (An  Acquisition  Decision  is  an  event 
that  yields  an  outcome  that  has  an  impact  on  another  organization  or  AC  AT  designated 
program  within  the  Acquisition  Lifecycle). 

9.  What  acquisition  decision(s)  does  your  organization  make  relative  to  the  decision 
product(s)? 

10.  What  is  the  intent  of  the  decision  product?  Does  the  decision  product  meet  its 
intended  purpose? 

1 1 .  What  information  products  do  you  believe  that  your  organization  should  receive 
(missing  information)  that  would  improve  your  acquisition  decision  making 
process? 

12.  What  information  products  does  your  organization  receive  that  are  non- value 
added  in  the  acquisition  decision  making  process? 

13.  What  Regulations  and  Policies  require  that  your  organization  creates  these 
products? 

14.  Do  you  use  an  IT  System  to  create/retrieve  a  decision  product?  If  so,  what  is  its 
name? 
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Survey  Administration 

The  survey  instrument  was  sent  to  the  participants  with  an  explanation  of  the 
research  being  conducted  and  how  their  assistance  would  help  with  the  project.  The 
participants  were  asked  to  respond  to  the  questions  and  send  the  information  back  to 
SAF/ACPO.  In  all  instances,  the  questions  were  either  answered  in  electronic  format  and 
returned  to  SAF/ACPO  via  email  or  answered  via  teleconference. 

Population  of  Interest 

The  sample  population  for  this  study  consisted  of  12  acquisition  decision-makers 
in  the  U.S.  Air  Force.  While  this  may  seem  like  a  relatively  low  sample,  it  is  much  larger 
than  was  expected  and  provides  a  great  deal  of  information  that  can  be  analyzed  from 
within  a  single  acquisition  organization.  The  population  size  of  the  ACAT  I  and  II 
acquisition  milestone  decision-makers  within  the  Air  Force  is  relatively  small.  Figure  8 
offers  a  view  of  the  number  of  Acquisition  Category  (ACAT)  IC  and  II  programs  with 
regard  to  the  milestone  decision  authorities  as  of  FY09.  ACAT  I  programs  are  estimated 
by  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 
(USD(AT&L))  to  require  eventual  expenditure  for  Research,  Development,  Test  and 
Evaluation  (RDT&E)  funding  of  more  than  $365  million  (Fiscal  Year  (FY)  2000  constant 
dollars)  or  procurement  of  more  than  $2.19  billion  (FY  2000  constant  dollars).  ACAT  II 
programs  are  estimated  by  the  DoD  Component  Head  to  require  eventual  expenditure  for 
RDT&E  of  more  than  $140  million  in  FY  2000  constant  dollars,  or  for  procurement  of 
more  than  $660  million  in  FY  2000  constant  dollars. 
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Figure  8.  USAF  Acquisition  Programs  by  MDA  Type  (FY09) 


The  number  of  Air  Force  AC  AT  IC  and  II  programs  is  17  and  25,  respectively.  Of  these 
42  programs,  the  12  participants  were  the  milestone  decision  authority  for  23  programs. 
What  is  also  important  is  that  the  information  obtained  from  these  participants  was 
complete  in  terms  of  answering  all  of  the  questions  in  the  questionnaire.  In  addition,  the 
participants  often  included  their  own  comments  or  opinions.  It  should  be  noted  that  the 
actual  people  completing  the  surveys  were  individuals  that  are  the  decision-makers 
within  their  respective  acquisition  organization  and  not  support  personnel.  All  of  the 
respondents  were  civilians  or  military  members  of  the  rank  0-6/GS-15  or  above. 


Data  Reduction  and  Content  Analysis 

Within  the  Miles  and  Huberman  (1994)  framework,  the  researcher  used  Microsoft 
Excel  as  a  means  for  data  reduction  to  analyze  content  for  all  three  sets  of  data.  The  first 
set  of  data  analyzed  was  the  program  knowledge  products  required  by  either  federal 
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statute  or  DoD  regulation  at  Key  Decision  Points.  The  initial  step  was  to  create  a  matrix 
that  captured  all  knowledge  products  and  indicated  whether  each  one  was  required  by 
statute  or  regulation;  the  matrix  also  indicated  to  which  Key  Decision  Point  the 
knowledge  product  is  applicable.  Table  6  shows  a  sample  of  the  matrix. 


Table  6.  Program  Knowledge  Product  Matrix 


ACAT  APPLICABILITY 

PROGRAM  KNOWLEDGE 

KEY  DECISION  POINT 
APPLICABILITY 

ID 

1C 

1AM 

IAC 

II 

III 

MDD 

A 

B 

c 

FRP 

R 

R 

R 

R 

R 

R 

ACQUISITION  STRATEGY  (AS) 

X 

X 

X 

R 

R 

R 

R 

R 

R 

ACQ  DECISION  MEMORANDUM  (ADM) 

X 

X 

X 

X 

R 

R 

R 

R 

R 

R 

ACQ  INFORMATION  ASSURANCE 
STRATEGY 

X 

X 

X 

X 

R 

R 

R 

R 

R 

R 

AFFORDABILITY  ASSESSMENTS 

X 

X 

S 

S 

S 

S 

S 

S 

ANALYSIS  OF  ALTERNATIVES  (AoA) 

X 

X 

X 

X 

The  next  data  reduction  technique  was  to  align  the  knowledge  products  to  one  (or 
more)  of  five  key  program  parameters:  Requirements,  Cost,  Schedule,  Technical 
Performance,  and  Funding.  This  enabled  the  researcher  to  group  the  knowledge  support 
products  according  to  the  area  in  which  they  provide  decision  support.  It  also  facilitated 
the  elimination  of  knowledge  products  that  did  not  support  one  of  the  aforementioned 
program  parameters.  Table  7  shows  a  sample  of  the  Knowledge  Products  and  their 
alignment  with  five  key  program  parameters. 
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Table  7.  Knowledge  Products  Alignment  with  Knowledge  Areas 


ACAT  APPLICABILITY 

PROGRAM  KNOWLEDGE 

APPLICABLE  KNOWLEDGE  AREA 

ID 

1C 

1AM 

IAC 

II 

III 

TECHNICAL 

REQUIREMENTS  COST  SCHEDULE 

PERFORMANCE 

FUNDING  STRATEGY 

R 

R 

R 

R 

R 

R 

CAPABILITY  PRODUCTION 

X  X 

DOCUMENT  (CPD) 

S 

S 

S 

S 

S 

S 

CLINGER-COHEN  ACT  (CCA) 
CONFIRMATION  COMPLIANCE 

X 

S 

S 

S 

S 

S 

S 

CLINGER-COHEN  ACT  (CCA) 
CERTIFICATION  COMPLIANCE 

X 

s 

S 

S 

S 

S 

CONSIDERATION  OF  TECHNOLOGY 

ISSUES 

X 

The  next  data  reduction  technique  was  performed  on  both  the  Knowledge 
Products  and  the  GAO’s  knowledge-based  decision-making  best  practices.  This 
technique  helped  the  researcher  align  the  knowledge  products  with  best  practices 
according  to  the  GAO:  technology  maturity,  design  stability,  and  production  maturity. 
Since  the  Knowledge  Points  are  limited  to  requirements,  design,  and  technical  aspects  of 
programs,  this  alignment  helped  the  researcher  further  reduce  the  number  of  knowledge 
products  to  use  in  the  research.  Table  8  shows  how  Knowledge  Products  are  aligned  with 
Knowledge  Points. 


Table  8.  Alignment  of  Knowledge  Products  and  Knowledge  Points 


PROGRAM  KNOWLEDGE 

APPLICABLE  KNOWLEDGE  AREA 

Alignment 

with 

Knowledge 

Alignment 

with 

Knowledge 

Alignment 

with 

Knowledge 

REQUIREMENTS  COST  SCHEDULE  FUNDING  STRATEGY 

PERFORMANCE 

Point  #1 

Point  #2 

Point  *3 

CAPABILITY  PRODUCTION 
DOCUMENT  (CPD) 

CLINGER-COHEN  ACT  (CCA) 
CONFIRMATION  COMPLIANCE 

CLINGER-COHEN  ACT  (CCA) 
CERTIFICATION  COMPLIANCE 

X  X 

X 

X 

X 

X 

CONSIDERATION  OF  TECHNOLOGY 

ISSUES 

X 

X 

X 

X 
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The  final  data  reduction  techniques  were  performed  on  the  answers  from  the 
questionnaire.  The  respondents  were  asked  14  total  questions,  but  only  three  questions 
were  applicable  to  this  research.  However,  the  respondents’  answers  to  the  remaining  1 1 
questions  were  taken  into  consideration  for  recommendations  made  at  the  conclusion  of 
this  research.  The  following  three  questions  were  used  for  this  research: 

1 .  What  is  the  intent  of  the  decision  product?  Does  the  decision  product  meet  its 
intended  purpose? 

2.  What  information  products  do  you  believe  that  your  organization  should 
receive  (missing  information)  that  would  improve  your  acquisition  decision¬ 
making  process? 

3.  What  information  products  does  your  organization  receive  that  are  non- value 
added  in  the  acquisition  decision-making  process 

The  answers  to  these  questions  were  analyzed  and  all  comments  that  referred  to  any 

decision  support  product  were  extracted  and  aligned  with  the  appropriate  decision 

product. 

For  the  first  question,  the  assumption  made  for  this  research  was  if  the  knowledge 
product  cannot  meet  its  intended  purpose,  then  it  also  cannot  effectively  support 
decisions  based  on  its  intended  purpose.  Question  la  (What  is  the  intent  of  the  decision 
product?)  was  not  used  unless  the  comments  supported  answers  given  in  other  questions. 
Question  lb  solicited  a  binary  response  (“YES”  or  “NO”  unless  the  respondent  left  this 
answer  blank)  and  the  responses  were  recorded  in  the  matrix.  Table  9,  on  the  following 
page,  shows  a  sample  of  the  respondents’  answers  to  Question  lb. 
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Table  9.  Knowledge  Products  Aligned  with  Survey  Results  (Question  lb) 


PROGRAM  KNOWLEDGE 

RMpondMt  #1 
Product  Center 

CV 

Respondent  M2 
DAO 

RHpndm  M 
DAO 

Respondent  i5 
SPM 

Respondent  *8 
SPM 

Rapotel  0 
SPM 

Respondent  it 
PEO 

Respondent  *9 
Wino  Director 

Rf.pcrdfi-  *10 

SPM 

Respondent  #1 1 
Wino  Director 

Respondent  *12 

WtaflCC 

01:  Mm*  * 
Intent? 

01:  Mm*  .1 
Intent? 

Ol  UmO  it 
Intent? 

Q1:HnM 

Intent? 

01.  Mm*  it 
Intent? 

Ol:  UmS  it 
Intent? 

01:  Meets  « 

Intent? 

01:  Mm*  It 
Intent? 

Ol:  Mm*  it 
Intent? 

Ol:  Mm*  it 
Intent? 

Ol  Mm*  it 
Intent? 

CAPABILITY  PRODUCTION 
DOCUMENT  (CPD) 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

YES 

CLINGER-COHEN  ACT  (CCA) 
CONFIRMATION  COMPLIANCE 

YES 

YES 

NO 

NO 

YES 

NO 

CLINGER-COHEN  ACT  (CCA) 
CERTIFICATION  COMPLIANCE 

NO 

NO 

YES 

NO 

CONSIDERATION  OF 

TECHNOLOGY  ISSUES 

NO 

NO 

NO 

NO 

NO 

NO 

NO 

NO 

Questions  2  and  3  offered  the  respondents  the  opportunity  to  share  which 
knowledge  products  they  thought  are  missing  and  those  which  are  non-value  added, 
respectively.  Question  2  captured  the  knowledge  products  that  could  improve  their 
decision-making  abilities.  There  were  very  few  responses  in  this  area.  The  researcher 
compared  this  list  of  knowledge  products  with  those  mandated  by  law  or  regulation.  If  it 
was  listed  and  required  by  law  or  regulation,  then  the  answer  was  ignored.  Question  3 
solicited  the  non-value  added  knowledge  products  in  the  acquisition  process.  If  any 
knowledge  product  was  listed  as  non-value  added  by  the  majority  of  the  respondents 
(seven  or  more)  and  not  required  by  law  or  regulation,  then  that  knowledge  product  was 
not  included  in  this  research. 

Comparison  of  Information  Types 

Miles  and  Huberman  (1994)  describe  data  reduction  activity  as  the  process  of 
choosing,  simplifying,  and  changing  the  data  that  is  obtained  in  the  research  process.  To 
analyze  the  data  and  help  answer  the  research  questions,  a  scoring  assessment  matrix  was 
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created.  This  assessment  focused  on  the  relationship  of  the  information  required  by 
statute  and  regulation  to  key  program  parameters:  requirements,  cost,  schedule, 
performance,  and  program  funding.  It  also  took  into  consideration  the  responses  from  the 
questionnaire  that  addressed  the  importance  of  the  decision  support  information. 
Assessment  Model 

The  assessment  was  accomplished  at  six  key  decision  points  in  the  acquisition 
process:  Materiel  Development  Decision,  Milestone  A,  Milestone  B,  Post  Critical 
Design  Review  -  A,  Milestone  C,  and  the  Full  Rate  Production  Decision  Review.  This 
assessment  used  a  scoring  schema  to  rate  the  decision  support  for  each  decision.  In 
addition  to  reducing  the  data,  this  research  altered  some  responses  without  changing  the 
intent  of  the  individual  surveyed.  This  was  accomplished  to  make  data  more 
manageable. 

The  assessment  model  is  based  on  four  criteria  statements.  Each  statement  has 
three  possible  criteria  responses.  To  assess  each  area  based  on  the  responses,  a  score  was 
assigned  to  each  response.  If  the  response  was  ‘YES’,  then  a  ‘+1’  was  scored  for  that 
particular  statement.  If  the  response  was  ‘NO’,  then  a  ‘-1’  was  scored  for  that  particular 
statement.  If  the  response  was  ‘Not  Applicable’,  then  a  ‘O’  was  scored  for  that  statement. 
Table  10  shows  the  statements,  responses,  and  corresponding  scores.  Table  1 1  shows  an 
example  of  the  scoring  schema  for  one  Decision  Support  Product  (DSP). 
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Table  10.  Knowledge  Assessment  Criteria  Statements  and  Responses 


Criteria  Statements 

Criteria  Responses 

Scoring 

Decision  support  product  is  required  by  Statute 
and/or  Regulation 

YES,  NO.  and  Not 
Applicable 

YE  S  -  +1 

Decision  support  product  is  aligned  with 

Knowledge  Point 

NO  =  -1 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

Not  Applicable  =  0 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

Table  1 1 .  Knowledge  Area  Assessment  Scoring  Example 


Knowledge  Area:  Program  Requirements 

Decision  Support:  Scope  Evolution 
Document  (changes  in  requirements) 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

-1 

-1 

-1 

-1 

-1 

-1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

0 

0 

0 

0 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

-1 

-1 

-1 

-1 

-1 

-1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

1 

1 

1 

1 

1 

1 

-1 

-1 

-1 

-1 

-1 

-1 

Decision  support  products  selected  for  the  knowledge  assessments  were  based  on 
the  products  most  frequently  used  by  the  decision-makers  responding  to  the 
questionnaire.  Each  decision  support  product  was  rated  in  accordance  with  the  score  for 
each  knowledge  area.  Each  decision  support  product  can  score  a  maximum  of  4  points 
and  a  minimum  of  -4  points,  where  positive  numbers  indicate  better  decision  support. 
Table  12  shows  the  scoring  schema  for  decision  support  products. 
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Table  12.  Scoring  Schema  for  Assessment  Model 


Score 

Assessment 

GREEN 

2.50  -  4.00 

Decision  Support  Product  is  available  and  provides 
adequate  information  to  make  informed  decision 

YELLOW 

0.01  -  2.49 

Decision  Support  Products  are  either  not  applicable  or 
more  information  is  needed  to  make  informed 
decision 

RED 

-4.00  -  0.00 

Additional  decision  support  is  needed 

The  last  step  in  using  the  model  is  to  assess  each  knowledge  area  based  on  the 
individual  decision  support  products.  This  research  used  a  quantitative  method  to  assign 
scores.  The  score  for  each  knowledge  area  was  computed  by  averaging  the  individual 
scores  of  the  decision  support  products.  Table  13  provides  an  example  of  a  knowledge 
area  assessment. 


Table  13.  Example  of  Knowledge  Area  Assessment 


Knowledge  Area:  Program  Requirements 

1.5 

Decision  Support :  Capabilities  Documents 

4 

Decision  Support:  Scope  Evolution 

-1 

The  next  activity  of  the  data  analysis  framework  is  verification  and  validity. 
Miles  and  Huberman  (1994)  stated  that  in  this  phase,  the  data  should  be  tested  for 
creditability  and  validility.  In  this  phase,  as  Miles  and  Huberman  (1994)  suggest,  before 
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drawing  conclusions  from  the  data,  the  research  must  again  consider  what  the  analyzed 
data  mean  and  then  assess  the  implications  toward  the  research  question.  In  reviewing 
the  data,  the  researcher  consistently  asked  the  following  question:  does  this  research 
method  make  sense?  In  this  case,  the  researcher  purposely  kept  the  methods  simple  and 
easy  to  understand.  Also,  any  assumptions  were  clearly  and  distinctly  indicated  in  the 
research. 

The  final  step  is  addressing  what  Miles  and  Huberman  (1994)  calls  the  pragmatic 
validity.  Although  formed  in  the  academic  setting,  qualitative  research  should  be  one 
that  can  be  extended  to  other  environments.  The  researcher  sought  to  focus  on  ensuring 
that  the  research  conducted  can  be  utilized  outside  of  the  academic  environment. 
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Chapter  IV.  Data  Analysis 


The  intent  of  this  chapter  is  to  present  the  analysis  of  the  data  gathered  during  this 
research.  As  previously  discussed,  the  data  was  derived  from  three  separate  sources.  All 
three  were  used  to  create  an  assessment  model  for  knowledge  areas  supporting  the  six 
major  decision  points  in  the  Department  of  Defense  (DoD)  acquisition  process.  Decision 
support  products  in  the  following  six  knowledge  areas  were  assessed:  program 
requirements,  cost,  schedule,  performance,  funding,  and  strategy.  The  knowledge 
support  assessment  results  were  based  on  the  statutory/regulatory  requirement,  answers 
provided  by  acquisition  decision  makers  with  regard  to  how  well  the  products  met  their 
intended  purpose  and  if  the  product  was  needed  to  make  decisions,  and  the  alignment  of 
the  knowledge  products  with  best  practices.  The  subsequent  sections  present  the  analysis 
in  greater  detail. 

Knowledge  Area:  Program  Requirements 

The  first  knowledge  area  assessed  in  this  research  was  program  requirements. 

The  Initial  Capabilities  Document  (ICD),  Capability  Development  Document  (CDD),  and 
Capability  Production  Document  (CPD)  are  the  primary  capability  documents  that 
provide  decision  support  in  this  knowledge  area.  These  documents  capture  the 
requirements  for  which  an  acquisition  program  is  based.  Decisions  made  in  the 
acquisition  process  should  support  filling  the  capability  gap  being  addressed  in  the 
capability  document.  Figure  9  shows  the  interrelationship  of  the  requirements  process 
and  the  acquisition  process. 
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=  Sponsor  Activity  ^^^JODSDocunent  Acquisition  decision 

Figure  9.  Interrelationship  of  the  Requirements  Process  and  the  Acquisition  Process 
(Defense  Acquisition  Guidebook,  2009) 

Capability  Documents 

Since  the  knowledge  contained  within  capability  documents  serve  as  the  basis  for 
acquisition  efforts,  it  was  an  area  that  was  expected  to  score  well  in  the  assessment. 
Capability  documents  are  required  by  regulation  at  every  major  decision  point  in  the 
acquisition  process.  Nine  respondents  indicated  that  requirements  documents  met  their 
intended  purpose.  However,  two  stated  that  it  is  not  always  clear  if  the  capability 
documents  (in  particular  the  Initial  Capability  Document)  are  compliant  with  the  DoD 
Architectural  Framework  (DoDAF).  The  DoDAF  is  a  reference  model  to  organize  the 
enterprise  architecture  (EA)  and  systems  architecture  into  complementary  and  consistent 
views.  In  addition,  although  compliancy  is  required,  the  benefit  of  being  compliant  is  not 
well  understood  in  the  acquisition  community.  In  their  comments,  the  respondents  stated 
that  the  acquisition  community  is  typically  not  an  active  participant  in  the  requirements 
generation  process  because  it  is  not  transparent  to  them  how  the  systems  being  acquired 
fit  into  the  overall  DoD  framework. 

Capability  documents  scored  an  average  of  4  (out  of  a  max  score  of  4)  for  this 
knowledge  area.  This  score  reflects  the  importance,  availability,  and  use  of  requirements 
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in  making  decisions  in  the  acquisition  process.  Requirements  documents  ranked  as  the 
number  one  decision  support  product  for  knowledge  support  in  this  research.  This 
indicates  that  the  products  are  available  and  provide  useful  information  that  is  used  to 
make  decisions  in  the  program  requirements  knowledge  area.  Table  14  shows  the  results 
of  the  scoring  in  by  each  decision  point. 


Table  14.  Capability  Documents  Knowledge  Assessment 


Knowledge  Area:  Program  Requirements 

Decision  Support:  Capabilities  Documents 
(ICD.CDD,  and/or  CPD) 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this 
knowledge  area  is  required  by  Statute 
and/or  Regulation 

1 

1 

1 

1 

1 

1 

Decision  support  product  is  aligned  with 
a  Knowledge  Point 

1 

1 

1 

1 

1 

1 

Decision  support  product  meets  its 
intended  purpose  per  respondents 

1 

1 

1 

1 

1 

1 

Decision  support  product  is  identified  as 
needed  information  by  respondents  to 
make  decisions 

1 

1 

1 

1 

1 

1 

4 

4 

4 

4 

4 

4 

Scope  Evolution 

Scope  evolution  is  the  next  decision  support  document  supporting  the  program 
requirement  knowledge  area.  Scope  evolution  documents  capture  changes  in  program 
requirements  prior  to  Milestone  B  (particularly  before  a  program  is  formally  baselined 
with  an  Acquisition  Program  Baseline  (APB)).  This  knowledge  support  was  referred  to 
by  nine  respondents.  All  nine  indicated  that  stability  of  capability  requirements 
(performance  or  quantity)  is  critical  to  program  success  as  changes  in  requirements 
impact  program  cost  and  schedule.  There  is  no  statutory  or  regulatory  document  required 
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for  a  scope  evolution  document.  However,  all  changes  to  Key  Performance  Parameters 
in  the  APB  (established  after  Milestone  B)  are  tracked  by  all  MDAP  programs. 

Three  respondents  indicated  an  issue  with  changes  in  requirements  prior  to 
Milestone  B.  The  issue  is  that  changes  to  requirements  after  a  program  has  started  are 
costly.  Additionally,  one  respondent  stated  that  the  “appetite  for  the  latest  and  greatest 
technology  is  eating  our  lunch,  when  a  lesser,  more  mature  technology  is  sufficient.” 

This  is  consistent  with  the  results  of  a  RAND  Corporation  study  conducted  in  2006.  For 
every  type  of  ship  they  studied,  the  price  escalation  rates  ranged  from  7  to  1 1  percent 
annually  between  1950  and  2000.  For  every  type  of  aircraft  that  was  examined,  the  price 
escalation  rates  ranged  from  7  to  12  percent  annually  between  1974  and  2005.  Since  the 
average  annual  inflation  rate  between  1965  and  2004  was  4.7%,  RAND  (2006)  concluded 
that  the  price  growth  above  the  inflation  index  stemmed  from  the  desire  for  more 
capabilities. 

Four  respondents  suggested  that  more  programs  follow  an  incremental  acquisition 
strategy  that  locks  program  requirements  at  Milestone  A.  They  went  on  to  state  that  any 
change  (increase)  in  requirements  should  be  acquired  through  separate  increments.  This 
approach  is  limited,  however,  to  the  manner  in  which  the  capabilities  are  required  by  the 
user.  For  example,  capability  documents  may  not  separate  requirements  into  increments, 
thus  forcing  the  acquisition  community  to  acquire  most  if  not  all  capabilities  for  a 
program  at  once. 

Scope  evolution  documents  scored  an  -1.  This  low  score  is  not  indicative  of  the 
value  of  the  knowledge  product,  but  rather  an  indication  that  the  knowledge  is  not 
required  by  statute  or  regulation.  This  score  indicates  that  more  knowledge  is  needed  to 
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support  decisions  in  this  knowledge  area.  These  decision  support  documents  are  aligned 


with  both  Knowledge  Point  1:  Requirements  and  Technology  Capability  are  Matched  and 


Knowledge  Point  2:  Design  will  work  as  required.  Table  15,  on  the  following  page, 


shows  the  scoring  of  this  assessment  area. 


Table  15.  Scope  Evolution  Knowledge  Assessment 


Knowledge  Area:  Program  Requirements 

Decision  Support:  Scope  Evolution 
Document  (changes  in  requirements) 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

-1 

-1 

-1 

-1 

-1 

-1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

0 

0 

0 

0 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

-1 

-1 

-1 

-1 

-1 

-1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

1 

1 

1 

1 

1 

1 

-1 

-1 

-1 

-1 

-1 

-1 

Knowledge  Area:  Program  Cost 

This  knowledge  area  focuses  on  program  cost  for  DoD  major  acquisition 
programs.  There  are  many  documents  used  by  the  individual  services  that  support  this 
knowledge,  including  the  initial  proposals  submitted  by  industry  partners.  In  addition, 
each  service  produces  a  cost  estimate  usually  referred  to  as  the  Service  Cost  Position. 
However,  this  research  focused  on  the  two  documents  that  are  mandated  by  statute  or 
regulation:  the  Independent  Cost  Estimate  and  the  Affordability  Assessment. 
Independent  Cost  Estimate 

The  Independent  Cost  Estimate  (ICE)  estimates  the  full  life-cycle  cost  for  a 
MDAP;  required  by  statute,  it  is  prepared  by  the  Director  of  Cost  Assessment  and 
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Program  Evaluation  (DC APE).  The  ICE  is  required  at  Milestones  A,  B,  and  C.  The  ICE 
was  commented  on  by  1 1  respondents.  All  indicated  that  the  document  supported 
decisions,  but  nine  acknowledged  that  cost  estimating  is  an  issue  for  MDAPs.  The  issue 
as  directly  stated  by  seven  respondents  is  that  costs  are  routinely  underestimated. 

These  statements  are  supported  by  a  2008  RAND  study  pertaining  to  the  sources 
of  cost  growth  in  MDAPs.  In  the  study  (RAND,  2008),  68  programs  were  examined  over 
the  previous  30  years.  They  found  that  after  adjusting  for  changes  in  the  quantity  of 
systems  produced,  costs  grew  by  46  percent  on  average  over  the  estimate  at  development 
approval  (Milestone  B).  Since  the  respondents  stated  that  this  decision  support  document 
was  historically  underestimated,  the  product  was  assessed  as  not  meeting  its  intent. 
Overall,  the  ICE  was  assessed  with  a  score  of  2.  Table  16  shows  the  scoring  of  this 
decision  support  product. 


Table  16.  Independent  Cost  Estimate  Knowledge  Assessment 


Knowledge  Area:  Cost  . =— 

Decision  Support: 

Independent  Cost  Estimate  (ICE) 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this 
knowledge  area  is  required  by  Statute 
and/or  Regulation 

0 

1 

1 

0 

1 

0 

Decision  support  product  is  aligned  with 
a  Knowledge  Point 

0 

1 

1 

0 

1 

0 

Decision  support  product  meets  its 
intended  purpose  per  respondents 

0 

-1 

-1 

0 

-1 

0 

Decision  support  product  is  identified  as 
needed  information  by  respondents  to 
make  decisions 

0 

i 

i 

0 

i 

0 

0 

2 

2 

0 

2 

0 

Affordability  Assessment 
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The  next  document,  which  is  required  at  Milestones  B  and  C,  assessed  in  the 
program  cost  knowledge  area  is  the  Affordability  Assessment.  The  intent  of  this 
document  is  to  demonstrate  that  the  program’s  projected  funding  and  manpower  are 
practical  and  attainable.  This  knowledge  support  area  was  addressed  by  ten  respondents, 
all  indicating  that  it  met  its  purpose.  Seven  respondents’  comments  were  focused  on  the 
staffing  level  of  program  offices  prior  to  Milestone  B,  which  is  when  major  acquisition 
efforts  are  designated  as  programs.  The  comments  suggested  that  Pre-Milestone  B 
programs  are  ill-equipped  to  be  successful  at  this  stage  because  the  staffing  level  is  too 
low  in  comparison  to  the  workload.  For  this  reason,  it  was  recommended  that  this 
document  be  mandated  for  Milestone  A.  Table  17  shows  the  scoring  of  this  decision 
support  product. 


Table  17.  Affordability  Assessment  Knowledge  Assessment 


Knowledge  Area:  Cost 

Decision  Support  Product: 
Affordability  Assessment 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

-1 

1 

0 

1 

0 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

0 

1 

0 

1 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

0 

1 

0 

1 

0 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

0 

1 

0 

1 

0 

0 

-1 

4 

0 

4 

0 

Knowledge  Area:  Program  Schedule 


This  research  assessed  two  key  documents  supporting  the  schedule  knowledge 


area.  The  first  document  is  the  Integrated  Master  Schedule  (IMS).  The  IMS  is  an 


integrated,  networked  schedule  containing  all  the  details  necessary  to  accomplish  the 
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work  specified  in  the  Integrated  Master  Plan  (IMP),  which  is  the  second  document 
assessed.  Since  both  documents  are  closely  related,  both  were  be  assessed  together. 

While  the  IMS  is  not  required  by  statute  or  regulation  as  a  standalone  document  in  the 
acquisition  process,  both  the  acquisition  strategy  and  the  APB  address  schedule 
parameters  for  MDAPs. 

Integrated  Master  Schedule 

All  12  respondents  commented  on  program  schedule.  Eleven  specifically  stated 
that  it  is  a  value-added  document  for  the  decision-making  process.  However,  four 
respondents  made  comments  indicating  that  the  government  is  too  dependent  on  the 
defense  partners  for  “detailed  schedule  information.”  Two  of  these  comments  were 
based  on  experiences  in  which  schedule  risks  were  not  reported  by  the  contractor  until 
they  became  issues  for  the  government.  Additionally,  five  respondents  indicated  that 
IMS  information  is,  at  times,  too  complex  to  be  used  to  make  decisions. 

Integrated  Master  Plan 

The  IMP  is  the  next  document  that  supports  the  schedule  knowledge  area.  Like 
the  IMS,  it  is  not  required  by  statute  or  regulation.  The  IMS  is  an  event-based  plan 
detailed  in  a  hierarchy  of  work  events.  None  of  the  respondents  made  comments 
specifically  with  regard  to  the  IMP.  However,  nine  made  mention  of  a  program’s  critical 
path,  which  was  determined  to  be  a  key  information  source  for  making  decisions.  Since 
the  IMP  contains  all  program  tasks,  this  research  inferred  that  the  respondents  deemed  the 
IMP  as  meeting  its  purpose.  Both  the  IMS  and  IMP  are  aligned  with  Knowledge  Point  3: 
Design  can  be  produced  within,  cost,  schedule,  and  quality  targets.  Table  18  shows  the 
scoring  for  these  decision  support  products. 


48 


Table  18.  Integrated  Master  Schedule/Plan  Knowledge  Assessment 


Knowledge  Area:  Schedule 

Decision  Support  Product: 
Integrated  Master  Schedule/Plan 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  reauired  bv  Statute  and/or  Regulation 

-1 

-1 

-1 

-1 

-1 

-1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

1 

1 

1 

1 

1 

1 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

1 

1 

1 

1 

1 

1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

1 

1 

1 

1 

1 

1 

2 

2 

2 

2 

2 

2 

Knowledge  Area:  Technical  Performance 

Six  decision  support  products  were  assessed  for  program  performance:  Systems 
Engineering  Plan  (SEP),  Consideration  for  Technology  Issues,  Test  and  Evaluation 
Strategy  (TES),  Independent  Technology  Readiness  Assessment  (ITRA),  Technical 
Readiness  Assessment  (TRA),  and  the  Test  and  Evaluation  Master  Plan  (TEMP).  All  six 
of  these  documents  are  required  by  either  statute  or  regulation.  For  purposes  of  this 
research,  the  ITRA  and  the  TRA  will  be  considered  the  same  document  and  scored 
similarly.  Similarly,  Test  and  Evaluation  Strategy  (TES)  and  the  Test  and  Evaluation 
Master  Plan  (TEMP)  were  considered  together. 

Systems  Engineering  Plan 

The  System  Engineering  Plan  (SEP)  is  the  next  knowledge  support  document 
assessed  for  this  research.  The  SEP’s  purpose  is  to  guide  programs  in  their  systems 
engineering  approach,  while  providing  a  documented  technical  foundation  for  the 
program.  It  documents  key  technical  risks,  processes,  resources,  and  metrics  associated 
with  the  program.  The  SEP  was  commented  on  by  all  12  respondents.  All  indicated  that 
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it  was  a  document  that  met  its  intent  in  supporting  decisions.  However,  only  three 
respondents  felt  that  the  SEP  provided  information  that  was  needed  to  make  decisions. 

Eight  respondents  felt  that  the  most  important  attribute  of  the  SEP  is  the  success 
criteria  for  the  technical  reviews.  These  comments  aligned  well  with  the  best  practices 
found  by  the  GAO  (Knowledge  Point  2:  Design  can  be  work  as  required),  which  uses  the 
percentage  of  engineering  drawings  completed  or  projected  for  completion  by  the  Critical 
Design  Review  as  one  critical  success  factor.  Four  respondents  made  comments 
regarding  the  standardization  of  the  SEP.  They  stated  that  the  lack  of  a  mandated  format 
made  it  difficult  to  use  for  making  program  decisions.  Table  19  shows  the  scoring  for 
this  knowledge  support  document. 


Table  19.  Systems  Engineering  Plan  Knowledge  Assessment 


Knowledge  Area:  Technical  Performance 

Decision  Support: 

Systems  Engineering  Plan 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

1 

1 

1 

1 

0 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

1 

1 

1 

1 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

1 

1 

1 

1 

0 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

-1 

-1 

-1 

-1 

0 

0 

2 

2 

2 

2 

0 

Consideration  for  Technology  Issues 

Consideration  for  Technology  Issues  is  the  second  decision  support  document 
assessed  for  this  area.  It  is  required  by  statute  at  Milestones  A  and  B.  The  law  mandates 
that  programs  provide  documentation  regarding  the  use  of  relevant  technologies. 
Therefore,  programs  must  address  whether  or  not  relevant  technologies  exist.  They  also 
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must  prove  the  rationale  for  choosing  to  not  use  technologies  if  they  exist.  Lastly,  the 
program  must  document  existing  relevant  technologies  that  will  be  incorporated  into  the 
program. 

Ten  respondents  commented  on  this  knowledge  support  product.  Eight  stated  that 
this  document  did  not  add  value  to  the  decision-making  process.  Of  those  eight,  six 
stated  that  the  intent  of  the  document  is  captured  in  other  required  documentation.  Two 
respondents  stated  that  they  did  not  understand  the  purpose  of  the  document.  As  a  result, 
this  document  was  considered  to  be  one  that  did  not  meet  its  intent.  Furthermore,  this 
knowledge  support  document  is  not  aligned  with  any  of  the  Knowledge  Points.  Table  20 
shows  the  scoring  for  this  area. 


Table  20.  Consideration  for  Technology  Issues  Knowledge  Assessment 


Knowledge  Area:  Technical  Performance 

Decision  Support  Product: 
Consideration  for  Technology  Issues 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

1 

1 

-1 

0 

0 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

-1 

-1 

-1 

0 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

-1 

-1 

-1 

0 

0 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

-1 

-1 

-1 

0 

0 

0 

■2 

■2 

■4 

0 

0 

Technology  Readiness  Assessment 

The  next  decision  support  documents  supporting  the  technical  performance  area 
were  the  Independent  Technology  Readiness  Assessment  (ITRA)  and  Technology 
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Readiness  Assessment  (TRA).  Both  documents  have  the  same  intent.  The  difference  is 
that  the  Weapons  Systems  Acquisition  Reform  Act  (WSARA)  of  2009  allows  the 
Director  of  Defense  Research  and  Engineering  (DDR&E)  to,  in  addition  to  the  TRA, 
require  an  additional  independent  assessment  of  the  technology  readiness. 

The  law  also  required  DDR&E  to  develop  a  knowledge-based  standard  against 
which  Technology  Readiness  Levels  (TRLs)  can  be  assessed.  The  law  mandated  the 
integration  of  risk  of  critical  technologies  at  key  stages  in  the  acquisition  process.  The 
TRA  is  required  at  Milestone  B  and  at  Milestone  C.  Nine  respondents  commented  on  the 
TRA,  with  all  comments  indicating  that  the  decision  support  document  met  its  intended 
purpose  and  that  the  information  provided  by  the  document  was  needed  to  make 
decisions.  Table  21  shows  the  scoring  for  these  documents. 


Table  21 .  Technology  Readiness  Assessment  Knowledge  Assessment 


Knowledge  Area:  Technical  Performance 

Decision  Support: 

Technology  Readiness  Assessment 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

-1 

1 

1 

1 

0 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

1 

1 

1 

1 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

1 

1 

1 

1 

0 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

1 

1 

1 

1 

0 

0 

2 

4 

4 

4 

0 

Test  and  Evaluation  Strategy /T est  and  Evaluation  Master  Plan 


The  last  two  documents  supporting  this  knowledge  area  are  the  Test  and 


Evaluation  Strategy  (TES)  and  the  Test  and  Evaluation  Master  Plan  (TEMP).  The  intent 


of  both  documents  is  very  similar.  The  TES  describes  the  approach  for  test  and 
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evaluation  for  programs  prior  to  Milestone  B.  The  TEMP,  which  evolves  from  the  TES, 
documents  the  overall  structure  and  objectives  for  the  test  and  evaluation  program 
beyond  Milestone  B.  Both  documents  are  required  by  regulation.  All  12  respondents 
made  comments  indicating  that  these  documents  met  their  respective  purpose.  However, 
only  five  indicated  that  they  were  needed  to  make  decisions.  Four  of  the  five  stated  that 
test  and  evaluation  results,  rather  than  the  plan  itself,  was  useful  in  making  decisions. 
These  documents  are  aligned  with  Knowledge  Point  2:  Design  will  work  as  required. 
Table  22  shows  the  assessment  for  these  documents. 


Table  22.  Test  and  Evaluation  Knowledge  Assessment 


Knowledge  Area:  Technical  Performance 

Decision  Support: 

Test  and  Evaluation  Master  Plan / 

Test  and  Evaluation  Strategy 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

1 

1 

1 

1 

1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

1 

1 

1 

1 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

1 

1 

1 

1 

1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

-1 

-1 

-1 

-1 

-1 

0 

2 

2 

2 

2 

1 

Knowledge  Area:  Funding 

The  next  knowledge  area  deals  with  program  funding.  Like  the  cost  knowledge 
area,  there  are  several  different  products  supporting  this  knowledge  area.  Most  of  the 
funding  products  produce  similar  information;  therefore,  this  research  focused  on  only 
two  decision  support  documents:  the  Milestone  Decision  Authority  (MDA)  certification 
at  Milestone  A  and  the  Non- Advocate  Cost  Assessment,  which  is  a  document  primarily 
used  by  the  Air  Force  acquisition  community. 
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Milestone  Decision  Authority  Certification 

The  MDA  certification  memorandum  is  required  by  the  Weapons  Systems 
Acquisition  Reform  Act  (WSARA)  of  2009  for  all  MDAPs  at  Milestone  A.  The 
WSARA  required  the  MDA  to  sign  a  memorandum  for  record  that  certifies  the  following 
program  attributes: 

1 .  Program  fulfills  an  approved  ICD,  executed  by  competent  entity. 

2.  Analysis  of  Alternatives  (AoA)  has  been  performed,  and  cost  estimate  is 
complete. 

3.  If  the  program  exceeds  25%  of  the  cost  or  schedule  target  prior  to  Milestone  B, 
then  program  termination  must  be  considered. 

At  the  time  of  the  survey,  the  WSARA  of  2009  was  not  signed  into  law. 

However,  all  12  respondents  had  knowledge  of  the  upcoming  law  and  provided 
comments.  For  this  reason,  the  answers  provided  may  have  been  skewed  since  their 
answers  may  not  have  been  based  on  factual  information  regarding  MDA  certification. 
All  12  respondents  felt  this  ‘Nunn-McCurdy’  like  process  was  being  conducted  too  early 
in  the  program  life.  One  respondent  stated  that  in  the  early  stages  of  the  acquisition 
process,  scope  is  still  being  defined  based  on  funding,  cost,  technology  maturity,  and 
initial  requirements;  therefore,  the  program  should  not  be  baselined.  Another  respondent 
said  this  will  lead  programs  to  artificially  inflate  their  cost  and  schedule  estimates  in 
order  to  avoid  breaching  the  threshold.  None  of  the  respondents  indicated  that  this 
document  added  any  value  to  the  decision-making  process.  This  document  is  aligned 
with  Knowledge  Point  3 :  Design  can  be  produced  within  cost,  schedule,  and  quality 
targets.  Table  23  shows  the  scoring  for  this  decision  support  document. 
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Table  23.  Milestone  Decision  Authority  Certification  Knowledge  Assessment 


Knowledge  Area:  Funding 

Decision  Support: 

MDA  Certification  (2366a) 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post- 

Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 

Production 

Decision 

Review 

Decision  product  supporting  this  knowledge  area 
is  required  bv  Statute  and/or  Regulation 

0 

1 

0 

0 

0 

0 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

1 

0 

0 

0 

0 

Decision  support  product  meets  its  intended 
ouroose  per  resoondents 

0 

-1 

0 

0 

0 

0 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

-1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Non-Advocate  Cost  Assessment 

The  Non-Advocate  Cost  Assessment  (NACA)  is  an  analysis  of  a  program’s  cost, 
price,  and  technical  risk.  The  primary  assessment  compares  the  cost  of  a  program  with 
the  program’s  FYDP  budget  and  assesses  whether  the  program  can  be  successful  within 
its  cost  targets.  It  is  prepared  by  an  independent  organization.  Ten  respondents  stated 
that  they  use  NACAs  when  making  decisions,  which  indicates  that  the  document  meets 
its  intended  purpose.  The  NACA  is  aligned  with  Knowledge  Point  3:  Design  can  be 
produced  within  cost,  schedule,  and  quality  targets.  Table  24  shows  the  scoring  for  this 
decision  support  product. 


Table  24.  Non- Advocate  Cost  Assessment  Knowledge  Assessment 


Knowledge  Area:  Funding 

Decision  Support: 

Non-Advocate  Cost  Assessment  (NACA) 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

-1 

-1 

-1 

-1 

-1 

-1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

1 

1 

1 

1 

1 

1 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

1 

1 

1 

1 

1 

1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

1 

1 

1 

1 

1 

1 

2 

2 

2 

2 

2 

2 
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Knowledge  Areas:  Program  Strategy 

The  decision  support  documents  supporting  this  knowledge  area  are  those  that 
provide  the  overall  strategy  that  will  achieve  the  cost,  schedule,  and  technical 
performance  objectives  of  the  program.  These  documents  are  the  Technology 
Development  Strategy  (TDS),  Acquisition  Strategy,  Acquisition  Program  Baseline 
(APB),  and  the  Life-Cycle  Support  Plan  (LCSP).  All  four  documents  are  required  by 
either  statute  or  regulation. 

Acquisition  Strategy 

The  first  decision  support  product  in  this  knowledge  area  is  the  Acquisition 
Strategy.  The  Acquisition  Strategy  is  the  all-inclusive  plan  that  provides  the  acquisition 
approach  and  describes  the  overall  strategy  for  the  program  management  team  to  manage 
risks  to  meet  program  goals.  The  Acquisition  Strategy  is  required  by  regulation  for 
Milestone  B  and  beyond.  This  document  was  commented  on  by  all  of  the  respondents 
that  it  met  its  purpose.  Additionally,  nine  made  comments  indicated  that  this  document 
provided  essential  knowledge  in  making  decisions.  This  document  is  aligned  with 
Knowledge  Point  3:  Design  can  be  produced  within  cost,  schedule,  and  quality  targets. 
Table  25  gives  the  assessment  score  for  the  Acquisition  Strategy. 
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Table  25.  Acquisition  Strategy  Knowledge  Assessment 


Knowledge  Area:  Program  Strategy 

Decision  Support: 

Acquisition  Strategy 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post- 

Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 

Production 

Decision 

Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

0 

1 

1 

1 

1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

0 

1 

1 

1 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

0 

1 

1 

1 

1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

0 

1 

1 

1 

1 

0 

0 

4 

4 

4 

3 

Technology  Development  Strategy 

The  TDS,  which  is  required  by  statute,  is  the  document  that  guides  acquisition 
efforts  in  the  Technology  Development  phase.  Its  function  is  very  similar  to  the 
Acquisition  Strategy.  All  12  respondents  offered  indications  that  it  met  its  intent  in 
supporting  the  decision-making  process.  Additionally,  seven  of  the  respondents  stated 
that  TDS  should  be  updated  more  frequently  within  the  Technology  Development  stage. 
Four  of  the  seven  reasoned  that  if  a  technology  is  proven  to  be  not  mature  enough  to  meet 
cost  and  schedule  parameters,  then  tradeoffs  need  to  be  made  as  soon  as  possible  to  avoid 
cost  and  schedule  overruns.  The  TDS  supports  Knowledge  Point  1 :  Requirements  and 
Technology  Capability  are  matched.  Table  26  shows  the  assessment  for  this  knowledge 
area. 
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Table  26.  Technology  Development  Strategy  Knowledge  Assessment 


Knowledge  Area:  Program  Strategy 

Decision  Support: 

Technology  Development  Strategy 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post- 

Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 

Production 

Decision 

Review 

Decision  product  supporting  this  knowledge  area 
is  reauired  bv  Statute  and/or  Reaulation 

0 

1 

0 

0 

0 

0 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

1 

1 

0 

0 

0 

0 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

1 

1 

0 

0 

0 

0 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

1 

1 

0 

0 

0 

0 

3 

4 

0 

0 

0 

0 

Acquisition  Program  Baseline 

The  next  assessed  document  which  supports  the  program  strategy  knowledge  area 
is  the  Acquisition  Program  Baseline  (APB).  The  establishment  of  program  goals  is 
required  by  statute  beginning  at  Milestone  B.  The  purpose  of  the  APB  is  to  document 
program  goals  in  terms  of  cost,  schedule,  and  technical  performance.  These  goals  are 
made  of  up  of  an  objective  value  and  a  threshold  value  for  each  Key  Performance 
Parameter  (KPP)  or  Key  System  Attribute  (KSA).  Eleven  of  the  12  respondents 
commenting  on  the  APB  indicated  that  it  added  value  to  the  decision-making  process, 
thus  meeting  its  intent.  Nine  respondents  made  comments  that  the  APB  is  needed  to 
make  decisions.  The  APB  is  aligned  with  Knowledge  Point  3:  Design  can  be  produced 
within  cost,  schedule,  and  quality  targets.  Table  27  provides  the  assessment  score  for  the 
APB. 
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Table  27.  Acquisition  Program  Baseline  Knowledge  Assessment 


Knowledge  Area:  Program  Parameters 

Decision  Support: 

Acquisition  Program  Baseline  (APB) 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

-1 

1 

1 

1 

1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

0 

1 

1 

1 

1 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

0 

1 

1 

1 

1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

0 

1 

1 

1 

1 

0 

-1 

4 

4 

4 

4 

Life  Cycle  Support  Plan 

The  last  document  supporting  the  program  strategy  area  is  the  Life  Cycle  Support 
Plan  (LCSP).  The  LCSP  documents  the  program’s  strategy  to  achieving  performance- 
oriented  product  support  capability.  The  LCSP  is  required  by  DoD  Directive  5000.01  to 
“implement  performance-based  logistics  strategies  that  optimize  total  system  availability 
while  minimizing  cost  and  logistics  footprint.”  Although  life-cycle  sustainment  planning 
begins  earlier  in  the  acquisition  process,  the  LCSP  is  mandated  by  Milestone  B. 

The  LCSP  was  commented  on  by  1 1  of  the  respondents,  indicating  that  it  met  its 
intent.  Eight  stated  that  the  LCSP  contained  information  that  could  be  useful  in  making 
decisions.  However,  three  acknowledged  that  the  information  in  the  LCSP  was  actually 
seldom  used  to  make  decisions.  Four  of  the  respondents  made  statements  suggesting  that 
logistical  planning  should  play  a  larger  role  in  the  overall  acquisition  strategy  for 
programs.  However,  they  felt  that  the  acquisition  community  lacks  the  right  skills  to 
effectively  plan  for  logistical  activities.  This  decision  support  document  does  not  directly 
align  with  any  of  the  GAO  Knowledge  Points.  Table  28  shows  the  assessment  for  the 
LCSP. 
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Table  28.  Life  Cycle  Support  Plan  Knowledge  Assessment 


Knowledge  Area:  Strategy 

Decision  Support: 

Life  Cycle  Support  Plan 

Materiel 

Development 

Decision 

Milestone  A 

Milestone  B 

Post-Critical 
Design 
Review  A 

Milestone  C 

Full  Rate 
Production 
Decision 
Review 

Decision  product  supporting  this  knowledge  area 
is  required  by  Statute  and/or  Regulation 

0 

0 

1 

1 

1 

1 

Decision  support  product  is  aligned  with  a 
Knowledge  Point 

0 

0 

-1 

-1 

-1 

-1 

Decision  support  product  meets  its  intended 
purpose  per  respondents 

0 

0 

1 

1 

1 

1 

Decision  support  product  is  identified  as  needed 
information  by  respondents  to  make  decisions 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

1 

1 

Overall  Assessment  of  Knowledge  Areas 

The  last  step  in  using  the  model  is  to  assess  each  knowledge  area  based  on  the 
individual  decision  support  products.  This  research  used  a  quantitative  method  by 
averaging  the  individual  scores  of  the  decision  support  products.  Table  29  shows  the 
assessment  results. 
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Table  29.  Assessment  of  Knowledge  Areas 


Knowledge  Area:  Program  Requirements 

1.5 

Decision  Support :  Capabilities  Documents 

4 

Decision  Support:  Scope  Evolution 

-1 

Knowledge  Area:  Cost 

2.165 

Decision  Support  Product:  Independent  Cost  Estimate 

2 

Decision  Support:  Affordability  Assessment 

2.33 

Knowledge  Area:  Schedule 

2 

Decision  Support:  Integrated  Master  Schedule 

2 

Decision  Support:  Integrated  Master  Plan 

2 

Knowledge  Area:  Technical  Performance 

2.54 

Decision  Support:  Systems  Engineering  Plan 

2 

Decision  Support:  Consideration  for  Technology  Issues 

2.67 

Decision  Support:  Technology  Readiness  Assessment 

3.5 

Decision  Support:  Test  and  Evaluation  Strategy/Test  and  Evaluation  Master 

2 

Knowledge  Area:  Funding 

1 

Decision  Support:  MDA  Certification  (2366a) 

0 

Decision  Support:  Non-Advocate  Cost  Assessment 

2 

Knowledge  Area:  Program  Strategy 

2.875 

Decision  Support:  Acquisition  Program  Baseline 

3.25 

Decision  Support:  Acquisition  Strategy 

3.75 

Decision  Support:  Technology  Development  Strategy 

3.5 

Decision  Support:  Life  Cycle  Sustainment  Plan 

1 
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Chapter  V.  Conclusions  and  Recommendations 


Cost  overruns  and  schedule  delays  have  plagued  Major  Defense  Acquisition 
Programs  (MDAPs)  within  the  Department  of  Defense  (DoD)  for  decades.  There  are 
several  factors  that  impact  an  acquisition  program’s  performance,  including  the  decisions 
made  by  defense  officials.  This  research  focused  on  one  aspect  of  decision-making: 
knowledge  support.  Secondary  interview  data,  assessments  of  knowledge  support 
documents,  and  how  these  documents  align  with  best  practices  for  knowledge-based 
decision-making  helped  address  the  investigative  questions  posed  by  this  research. 

Results 

This  research  posed  four  investigative  questions.  By  exploring  previous  research, 

gathering  and  analyzing  data,  and  drawing  conclusions,  this  research  has  answered  those 

questions.  The  research  outcomes  are  presented  for  each  question. 

What  are  the  critical  pieces  of  information  that  are  required  for  an  informed  decision  at 
key  decision  points  in  the  acquisition  process? 

The  research  answered  this  investigative  question  by  establishing  which 
documents  were  needed  to  make  decisions.  The  required  information,  according  to  the 
respondents,  was  considered  to  be  stated  in  products  that  adequately  addressed  one  of  six 
key  knowledge  areas  of  an  acquisition  program:  Requirements,  Cost,  Schedule, 

Technical  Performance,  Funding,  and  Strategy.  This  research  focused  on  17  decision 
support  documents  that  directly  supported  these  knowledge  areas.  However,  based  on  the 
feedback  from  respondents,  two  documents  were  eliminated.  The  Milestone  Decision 
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Authority  (MDA)  Certification  and  the  Consideration  for  Technology  Issues  were  not 
considered  value-added  to  the  decision-making  process.  Table  30  shows  the  list  of 
critical  decision  support  products. 


Table  30.  Critical  Decision  Support  Products 


Capabilities  Documents 

Scope  Evolution 

Independent  Cost  Estimate 

Affordability  Assessment 

Integrated  Master  Schedule 

Integrated  Master  Plan 

Systems  Engineering  Plan 

Consideration  for  Technology  Issues 

Technology  Readiness  Assessment 

Test  and  Evaluation  Strategy 

Test  and  Evaluation  Master  Plan 

Non-Advocate  Cost  Assessment 

Acquisition  Program  Baseline 

Acquisition  Strategy 

Technology  Development  Strategy 

Life  Cycle  Support  Plan 

How  does  the  currently  available  information  (contained  in  required  decision  products) 
compare  to  what  is  needed  for  decisions  with  regard  to  technology,  design,  and 
production  knowledge  for  each  of  the  key  decision  points  in  the  acquisition  process? 

This  investigative  question  dealt  with  the  alignment  of  the  DoD  acquisition 
process  with  best  practices  identified  by  the  General  Accounting  Office  (GAO)  (2008). 
The  research  indicated  that  there  is  in  fact  an  alignment  of  knowledge  mandated  with  the 
GAO  Knowledge  Points.  For  every  Knowledge  Point,  there  are  decision  support 
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products  required  by  law  or  regulation  aligned  to  it.  Additionally,  this  research  has 
concluded  that  alignment  with  this  best  practice  has  not  always  translated  into  better 
program  performance. 

Best  practices  in  decision  support  outside  of  the  knowledge  products  should  also 
be  considered.  For  example,  one  metric  the  GAO  uses  to  assess  program  knowledge  is 
the  average  percent  of  design  drawings  completed  by  the  Critical  Design  Review. 
Although  readily  available,  this  data  is  not  required  by  statute  or  regulation  for  any 
decision  support  product.  Since  the  format  of  most  required  documents  is  not  dictated,  it 
is  possible  that  this  data  is  captured.  Therefore,  the  alignment  of  the  knowledge  cannot 
be  determined  conclusively. 

What  is  the  effect  of  lack  of  key  information  on  DoD  acquisition  programs? 

This  research  defined  key  information  on  DoD  acquisition  programs  in  two  ways. 
The  first  were  the  instances  in  which  key  information  is  defined  by  best  practices.  As 
previously  mentioned,  the  GAO  (2008)  uses  metrics  to  assess  program  knowledge.  As 
such,  knowledge  supporting  these  metrics  can  be  considered  key  information  per  best 
practices.  However,  this  research  did  not  specifically  address  the  alignment  of  decision 
support  documents  and  the  information  supporting  the  metrics.  Therefore,  the  effect  of 
the  lack  of  key  information  supporting  the  GAO  metrics  is  unknown. 

The  second  definition  of  key  information  is  the  knowledge  contained  in  decision 
support  products  determined  by  decision-makers  as  being  needed  to  make  decisions.  In 
this  regard,  if  key  information  is  missing,  then  the  effect  is  simply  that  decision-makers 
cannot  make  a  well-informed  decision.  As  an  example,  the  research  identified  the  Scope 
Evolution  Document  as  key  missing  information.  At  least  one  comment  from  the 
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respondents  indicated  that  information  in  this  document  could  be  used  to  make  better 
decisions  regarding  tradeoffs  in  terms  of  cost,  schedule,  and  technical  performance. 

Can  this  effect  be  quantified  in  terms  of  program  cost,  schedule,  and  technical 
performance? 

It  is  not  clear  if  missing  information  can  be  quantified  in  terms  of  cost,  schedule, 
and  technical  performance  as  there  are  so  many  factors  that  impact  these  program 
parameters.  The  literature  review  did  reveal  that  National  Aeronautics  and  Space 
Administration  (NASA)  implemented  policy  supporting  the  GAO’s  best  practices  in 
knowledge-based  decision-making  and  the  results  showed  there  were  no  improvements  in 
cost  and  schedule  for  those  projects. 

Recommendations 

Based  on  the  information  that  has  been  reviewed,  it  is  possible  to  make  several 
recommendations  to  improve  the  use  of  knowledge-based  decision-making  in  DoD 
acquisitions.  These  recommendations  are  based  directly  on  the  information  obtained 
from  the  participants  completing  the  surveys  used  in  this  study,  as  well  as  from  the 
information  obtained  by  reviewing  literature.  It  is  important  to  note  that  these 
recommendations  are  not  intended  to  imply  that  the  DoD  acquisition  process  is 
completely  inefficient  or  should  be  completely  changed.  Instead,  these  recommendations 
are  made  as  a  way  to  improve  the  decision-making  process  and  to  bring  about  greater 
efficiency  in  the  work  that  is  done  to  acquire  new  systems  for  the  DoD  by  its  acquisition 
teams,  suppliers,  and  other  stakeholders. 

Recommendation  #1 
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The  first  recommendation  is  to  further  this  study  by  exploring  ways  to  improve 
the  score  of  the  knowledge  areas.  This  can  be  accomplishing  by  focusing  on  the  drivers 
for  the  low  scores  for  decision  support  products.  For  example,  the  Non- Advocate  Cost 
Assessment  (NACA)  scored  well  in  all  areas  with  the  exception  of  it  not  being  mandatory 
for  major  reviews.  In  this  case,  it  means  that  it  is  considered  valuable  for  decision¬ 
makers  but  it  has  not  been  mandated.  By  mandating  this  knowledge  product,  it  ensures 
that  all  decision-makers  will  have  access  to  valuable  knowledge  when  making  decisions. 
Recommendation  #2 

The  second  recommendation  is  to  use  specific  program  metrics  at  reviews  to 
support  decisions.  For  example,  one  metric  the  GAO  recommends  for  the  Critical  Design 
Review  is  the  percentage  of  design  drawings  that  are  complete.  This  metric  captures  the 
stability  of  the  design.  The  literature  review  indicated  that  the  higher  the  percentage  of 
drawings  that  are  complete,  the  higher  the  probability  of  the  design  being  capable  of 
meeting  performance  requirements.  Additionally,  a  stable  design  in  the  early  stages  of 
the  development  cycle  reduces  the  risk  of  design  changes.  The  respondents  to  the  survey 
also  recommend  another  metric:  changes  in  requirements  (Scope  Evolution).  This 
metric  captures  the  stability  of  requirements,  which  helps  keeps  cost  and  schedule  within 
goals. 

Recommendation  #3 

The  next  recommendation  is  that  decision  authorities  should  have  more  freedom 
to  request  specific  types  of  decision  products.  Aside  from  the  decision  products  that  are 
required  by  statute  and  regulation,  decision  authorities  should  have  the  freedom  to 
request  additional  decision  products  when  they  believe  that  the  information  contained  in 
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those  products  would  be  useful  for  a  key  decision.  This  freedom  to  have  access  to 
additional  decision  products,  or  even  decision  products  that  are  not  routinely  used  or 
made  available  to  acquisition  teams,  would  help  to  overcome  the  feelings  that  exist  about 
the  lack  of  information  when  making  a  major  decision  for  a  program. 

In  addition  to  the  decision  authority,  allowing  other  functional  members  of  the 
review  board  to  specify  decision  products  would  also  be  of  benefit.  Rather  than  decision 
products  being  shared  between  stakeholders  that  are  viewed  as  being  useless  and  not  even 
being  thoroughly  read,  or  even  read  at  all,  the  ability  to  request  specific  decision  products 
would  likely  result  in  all  stakeholders  taking  the  documents  that  they  receive  more 
seriously  and  using  the  information  contained  in  those  products  in  a  more  efficient 
manner. 

Recommendation  #4 

The  next  recommendation  is  to  mandate  a  template  for  use  during  decision 
briefings  of  major  milestones.  This  template  would  contain  a  specific  recommendation 
for  the  milestone  decision  supported  by  knowledge  within  the  mandated  knowledge 
support  product.  For  example,  the  senior  engineering  functional  would  make  a  specific 
recommendation  to  the  decision  authority  with  regard  to  technical  performance.  This 
recommendation  would  be  based  on  the  knowledge  gained  from  documents  like  the 
Systems  Engineering  Plan  and  Independent  Technology  Readiness  Document.  This 
would  accomplish  two  important  tasks.  First,  it  would  ensure  the  decisions  are  being 
based  on  knowledge  that  is  available  and  that  mandated  knowledge  products  are  being 
reviewed.  Secondly,  it  would  ensure  that  the  decision  support  product  is  being 
thoroughly  reviewed  for  knowledge  that  can  be  used  to  make  program  decisions. 
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Recommendation  # 5 


Finally,  it  is  recommended  that  the  DoD  create  a  standardized  means  by  which 
decision  products  across  the  department  can  be  created  and  stored  for  use  by  all 
programs,  regardless  of  Acquisition  Category  (ACAT)  level.  One  of  the  issues  raised  by 
data  collected  from  the  participants  in  this  study  is  that  there  is  no  standardized  means  by 
which  to  create  and  store  decision  products.  Instead,  information  is  simply  received  and 
stored  in  whatever  way  a  specific  team  member  may  believe  is  best.  This  creates  a 
situation  where  efficiencies  cannot  be  realized  because  programs  are  unable  to  share 
information  used  to  make  decisions. 

The  creation  or  adoption  of  a  single  Information  Technology  system  to 
standardize  the  process  of  creating  and  storing  decision  products  would  help  make  the 
use  of  decision  products  more  engrained  in  the  culture  of  the  DoD.  Having  a 
standardized  process  to  handle  decision  products  would  be  an  indication  to  acquisition 
teams  that  decision  products  are  no  longer  viewed  as  something  that  can  be  read  and  then 
forgotten  about  in  the  future.  Instead,  decision  products  could  be  collected  and  team 
members  could  easily  access  the  information  contained  in  those  products  and  gain  a 
feeling  that  the  products  are  genuinely  important  and  should  be  taken  seriously. 

Limitations 

As  with  any  research  effort,  this  effort  has  limitations.  The  respondents  to  the  survey 
represented  only  the  service’s  acquisition  community.  Secondly,  there  is  little 
established  research  in  this  area.  As  such,  the  researcher  does  not  have  a  point  of 
reference  for  the  findings.  Additionally,  since  some  data  sets  were  limited  to  the  Air 
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Force,  the  findings  may  not  be  applicable  to  other  organizations  within  the  DoD. 

Another  limitation  is  the  fact  that  the  researcher  had  to  interpret  the  meaning  of  some 
responses.  This  involved  a  combination  of  deductive  and  inductive  analysis,  which  may 
or  may  not  have  led  to  the  true  meaning  of  the  submission  by  the  individuals  surveyed. 
The  final  limitation  is  the  aforementioned  sample  size  of  12.  Statistically,  it  is  not  a 
significant  number,  but  it  represents  a  significant  portion  of  the  milestone  decision 
authorities  for  the  Air  Force. 

Summary 

Decision-making  for  acquisition  programs  is  very  important.  The  decisions  that 
are  made  not  only  affect  the  performance  of  a  program  but  could  also  impact  the  lives  of 
Airman,  Soldiers,  Sailors,  and  Marines  protecting  our  country.  Analyzing  the  decision 
support  products  is  one  method  to  improve  the  knowledge  that  is  used  to  make  decisions. 
If  these  recommendations  are  put  into  place,  the  researcher  believes  that  the  acquisitions 
process  within  the  DoD  would  indeed  be  improved.  While  these  recommendations  may 
seem  relatively  simplistic,  they  are  based  on  actual  issues  that  have  been  addressed  by 
active  members  of  Air  Force  acquisition  teams.  By  working  to  improve  the  efficiency  of 
the  decision-making  process,  it  will  lead  to  better  decisions  for  programs. 
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Appendix  A:  Statutory  Requirements  for  Milestone  A 


INFORMATION  REQUIRED  FOR 

MILESTONE  A 

APPLICABLE  STATUTE,  POLICY,  AND 

OTHER  REFERENCES 

Analysis  of  Alternatives 

Subtitle  III  of  title  40,  U.S.C. 

Section  2366a  of  Title  10,  U.S.C. 

Clinger-Cohen  Act  Compliance 
(All  IT-including  National 

Security  Systems  (NSS)) 

Subtitle  III  of  title  40,  U.S.C. 

Consideration  of  Technology 

Issues 

Paragraph  (b)(5)  of  Section  2364  of  Title 

10,  U.S.C. 

Cooperative  Opportunities 
(part  of  TDS) 

Paragraph  (e)  of  Section  2350a  of  Title  10, 
U.S.C. 

Data  Management  Strategy 
(part  of  TDS) 

Section  2320  of  Title  10,  U.S.C. 

Independent  Cost  Estimate 

Section  2434  of  Title  10,  U.S.C. 

Market  Research 

Section  2377  of  Title  10,  U.S.C. 

Paragraph  (e)(2)  of  Section  644  of  title  15, 
U.S.C. 

Milestone  Decision  Authority 
Program  Certification 

Section  2366a  of  Title  10,  U.S.C. 

DoD  Instruction  5000.02 

Submission  of  a  DD  Form  1494 
and  Certification  of  Spectrum 
Support 

Sections  305  and  901  through  904  of  title 
47,  U.S.C. 

Section  104  ofP.L.  102-538 

Part  2  of  OMB  Circular  A- 1 1 

Technology  Development  Strategy 

Section  803  of  Public  Law  107-314 
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Appendix  B:  Regulatory  Requirements  for  Milestone  A 


INFORMATION  REQUIRED 

FOR  MILESTONE  B 

APPLICABLE  REGULATION, 
POLICY,  AND  OTHER 

REFERENCES 

Acquisition  Decision 

Memorandum 

DoD  Instruction  5000.02 

Acquisition  Information 

Assurance  Strategy  (All  IT- 
including  NSS) 

DoD  Instruction  8580.1 

DoD  Instruction  5000.02 

DoD  Component  Cost  Estimate 
(as  required  by  Component 
Acquisition  Executive  for  MDAP) 

DoD  Instruction  5000.02 

Exit  Criteria 

DoD  Instruction  5000.02 

Initial  Capabilities  Document 

CJCS  Instruction  3170.01 

Item  Unique  Identification 
Implementation  Plan 

DoD  Instruction  8320.04 

Life-Cycle  Signature  Support  Plan 

DoD  Directive  5250.01 

Net-Centric  Data  Strategy 
(Approach  summarized  in  TDS) 

DoD  Directive  8320.02 

Program  Protection  Plan 

DoD  Instruction  5200.39 

Systems  Engineering  Plan 

DoD  Instruction  5000.02 

Test  and  Evaluation  Strategy 

DoD  Instruction  5000.02 
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Appendix  C:  Statutory  Requirements  for  Milestone  B 


INFORMATION  REQUIRED 
FOR  MILESTONE  B 

APPLICABLE  STATUTE,  POLICY, 

AND  OTHER  REFERENCES 

Acquisition  Program  Baseline 
(APB) 

Section  2435  of  title  10,  U.S.C. 

Alternate  Live  Fire  Test  and 
Evaluation  (LFT&E)  Plan 

Section  2366  of  title  10,  U.S.C. 

Analysis  of  Alternatives 

Subtitle  III  of  title  40,  U.S.C. 

Section  2366a  of  title  10,  U.S.C. 

Benefit  Analysis  and 

Determination 

Paragraph  (e)  of  Section  644  of  title  15, 
U.S.C. 

Clinger-Cohen  Act  (CCA) 
Compliance 

Subtitle  III  of  title  40,  U.S.C. 

Competition  Analysis 
(part  of  Acquisition  Strategy) 

Section  2469  of  title  10,  U.S.C. 

Consideration  of  Technology 

Issues 

Paragraph  (b)(5)  of  Section  2364  of  Title 

10,  U.S.C. 

Cooperative  Opportunities 
(part  of  Acquisition  Strategy) 

Paragraph  (e)  of  Section  2350a  of  title  10, 
U.S.C. 

Core  Logistics  Analysis/Source  of 
Repair  Analysis  (part  of 
Acquisition  Strategy) 

Section  2464  of  title  10,  U.S.C. 

Section  2466  of  title  10,  U.S.C. 

Data  Management  Strategy 
(part  of  Acquisition  Strategy) 

Section  2320  of  title  10,  U.S.C. 
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Independent  Cost  Estimate  (ICE) 

Section  2434  of  title  10,  U.S.C. 

Industrial  Base  Capabilities 
Considerations 

Section  2440  of  title  10,  U.S.C. 

LFT&E  Waiver  from  Full-up, 
System-level  Testing 

Section  2366  of  title  10,  U.S.C. 

Low-Rate  Initial  Production 
Quantities 

Section  2400  of  title  10,  U.S.C. 

Manpower  Estimate 

Section  2434  of  title  10,  U.S.C. 

Market  Research 

Section  2377  of  Title  10,  U.S.C. 

Paragraph  (e)(2)  of  Section  644  of  title  15, 
U.S.C. 

Milestone  Decision  Authority 
Program  Certification 

Section  2366b  of  title  10,  U.S.C. 

DoD  Instruction  5000.02 

Military  Equipment  Valuation 

Public  Law  101-576,  SFFAS  6 

Programmatic  Environmental, 
Safety  and  Occupational  Health 
Evaluation  (PESHE) 

Sections  4321-4347  of  title  42,  U.S.C. 

E.O.  12114 

Replaced  System  Sustainment  Plan 

Section  2437  of  title  10,  U.S.C. 

Selected  Acquisition  Report 

Section  2432  of  title  10,  U.S.C. 

Section  2445d  of  title  10,  U.S.C. 

Submission  of  a  DD  Form  1494 
and  Certification  of  Spectrum 
Support 

Sections  305  and  901-904  of  title  47, 

U.S.C. 

Section  104  of  Public  Law  102-538 

Part  2  of  OMB  Circular  A- 1 1 
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Appendix  D:  Regulatory  Requirements  for  Milestone  B 


INFORMATION  REQUIRED  FOR 
MILESTONE  B 

APPLICABLE  REGULATION, 
POLICY,  AND  OTHER 

REFERENCES 

Acquisition  Decision  Memorandum 

DoD  Instruction  5000.02 

Acquisition  Information  Assurance 
Strategy 

DoD  Instruction  8580.1 

DoD  Instruction  5000.02 

Acquisition  Strategy 

DoD  Instruction  5000.02 

Affordability  Assessment 

DoD  Instruction  5000.02 

Capability  Development  Document 

CJCS  Instruction  3170.01 

Chief  Information  Officer 

Confirmation  of  CCA  Compliance 

DoD  Instruction  5000.02 

Corrosion  Plan 

DoD  Instruction  5000.67 

DoD  Instruction  5000.02 

Cost  Analysis  Requirements 
Description  (CARD) 

DoD  Instruction  5000.02 

Defense  Acquisition  Executive 
Summary 

DoD  Instruction  5000.02 

DoD  Component  Cost  Estimate 

DoD  Instruction  5000.02 

Exit  Criteria 

DoD  Instruction  5000.02 

Technology  Readiness  Assessment 

DoD  Instruction  5000.02 
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Information  Support  Plan  (ISP)  (All 
IT-including  NSS) 

DoD  Directive  4630.05 

DoD  Instruction  4630.8 

Initial  Capabilities  Document 

CJCS  Instruction  3170.01 

Item  Unique  Identification 
Implementation  Plan 

DoD  Instruction  8320.04 

Life-Cycle  Signature  Support  Plan 

DoD  Directive  5250.01 

Life-Cycle  Sustainment  Plan 
(part  of  Acquisition  Strategy) 

DoD  Instruction  5000.02 

Net-Centric  Data  Strategy  (Approach 
detailed  in  ISP) 

DoD  Directive  8320.02 

Operational  Test  Agency  Report  of 
Operational  Test  and  Evaluation 

Results 

DoD  Instruction  5000.02 

Preliminary  Design  Review  Report 

DoD  Instruction  5000.02 

Program  Protection  Plan 

DoD  Instruction  5200.39 

Spectrum  Supportability 

Determination 

DoD  Directive  4650.1 

System  Threat  Assessment  Report 
(STAR) 

DoD  Instruction  5000.02 

DoD  Directive  5105.21 

Systems  Engineering  Plan 

DoD  Instruction  5000.02 

Technology  Readiness  Assessment 

DoD  Instruction  5000.02 

Test  and  Evaluation  Master  Plan 

DoD  Instruction  5000.02 
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Appendix  E:  Statutory  Requirements  for  Milestone  C 


INFORMATION  REQUIRED 

FOR  MILESTONE  C 

APPLICABLE  STATUTE,  POLICY, 

AND  OTHER  REFERENCES 

Acquisition  Program  Baseline 
(APB) 

Section  2435  of  title  10,  U.S.C. 

Analysis  of  Alternatives 

Subtitle  III  of  title  40,  U.S.C. 

Section  2366a  of  title  10,  U.S.C. 

Benefit  Analysis  and 

Determination  (applicable  to 
bundled  acquisitions  only  if  no  MS 
B))  (part  of  Acquisition  Strategy) 

Paragraph  (e)  of  Section  644  of  title  15, 
U.S.C. 

Clinger-Cohen  Act  (CCA) 
Compliance 

(All  IT-including  National 

Security  Systems  (NSS)) 

Subtitle  III  of  title  40,  U.S.C. 

Competition  Analysis 
(Depot-level  Maintenance  $3M 
rule) 

(part  of  Acquisition  Strategy) 

(If  No  MS  B) 

Section  2469  of  title  10,  U.S.C. 

Consideration  of  Technology 

Issues 

Paragraph  (b)(5)  of  Section  2364  of  Title 

10,  U.S.C. 

Cooperative  Opportunities 
(part  of  Acquisition  Strategy) 

Paragraph  (e)  of  Section  2350a  of  title  10, 
U.S.C. 

Core  Logistics  Analysis/Source  of 
Repair  Analysis  (part  of 
Acquisition  Strategy) 

Section  2464  of  title  10,  U.S.C. 

Section  2466  of  title  10,  U.S.C. 

Data  Management  Strategy 

Section  2320  of  title  10,  U.S.C. 

76 


Independent  Cost  Estimate  (ICE) 

Section  2434  of  title  10,  U.S. C. 

Industrial  Base  Capabilities 

Considerations 

(part  of  Acquisition  Strategy) 

Section  2440  of  title  10,  U.S. C. 

Manpower  Estimate  (reviewed  by 
the  office  of  the  Under  Secretary  of 
Defense  for  Personnel  and 
Readiness) 

Section  2434  of  title  10,  U.S.C. 

Milestone  Decision  Authority 
Program  Certification  (If  Program 
Initiation) 

Section  2366a  of  title  10,  U.S.C. 

Section  2366b  of  title  10,  U.S.C. 

DoD  Instruction  5000.02 

Military  Equipment  Valuation 
(part  of  Acquisition  Strategy) 

Public  Law  101-576 

Statement  of  Federal  Financial  Accounting 
Standards  No.  6 

Programmatic  Environmental, 
Safety  and  Occupational  Health 
Evaluation  (PESHE)  (Including 
National  Environmental  Policy  Act 
(NEPA)  /  Executive  Order  (E.O.) 
12114  Compliance  Schedule) 

Sections  4321-4347  of  title  42,  U.S.C. 

E.O.  12114 

Submission  of  a  DD  Form  1494 
and  Certification  of  Spectrum 
Support 

(applicable  to  all 
systems/equipment  that  use  the 
electromagnetic  spectrum  while 
operating  in  the  U.S.  and  its 
possessions) 

Sections  305  and  901-904  of  title  47, 

U.S.C. 

Section  104  of  Public  Law  102-538 

Part  2  of  OMB  Circular  A-l  1 
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Appendix  F:  Regulatory  Requirements  for  Milestone  C 


INFORMATION  REQUIRED 
FOR  MILESTONE  C 

APPLICABLE  REGULATION, 
POLICY,  AND  OTHER 

REFERENCES 

Acquisition  Decision 

Memorandum 

DoD  Instruction  5000.02 

Acquisition  Information  Assurance 
Strategy  (All  IT-including 

National  Security  Systems  (NSS)) 

DoD  Instruction  8580.1 

DoD  Instruction  5000.02 

Acquisition  Strategy 

DoD  Instruction  5000.02 

Affordability  Assessment 

DoD  Instruction  5000.02 

Capability  Development  Document 

CJCS  Instruction  3170.01 

Chief  Information  Officer 
Confirmation  of  CCA  Compliance 

DoD  Instruction  5000.02 

Corrosion  Plan 

DoD  Instruction  5000.67 

DoD  Instruction  5000.02 

Cost  Analysis  Requirements 
Description  (CARD) 

DoD  Instruction  5000.02 

Defense  Acquisition  Executive 
Summary 

DoD  Instruction  5000.02 

DoD  Component  Cost  Estimate 

DoD  Instruction  5000.02 

Exit  Criteria 

DoD  Instruction  5000.02 

Independent  Technology 

Readiness  Assessment 

DoD  Instruction  5000.02 
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(if  required  by  the  office  of  the 
Director,  Defense  Research  and 
Engineering) 

Information  Support  Plan  (ISP)  DoD  Directive  4630.05 
(All  IT-including  NSS)  DoD  Instruction  4630.8 

Initial  Capabilities  Document  CJCS  Instruction  3170.01 

Item  Unique  Identification  DoD  Instruction  8320.04 

Implementation  Plan 

Life-Cycle  Signature  Support  Plan  DoD  Directive  5250.01 

Life-Cycle  Sustainment  Plan  DoD  Instruction  5000.02 
(part  of  Acquisition  Strategy) 

Net-Centric  Data  Strategy  DoD  Directive  8320.02 

(Approach  detailed  in  ISP) 

Operational  Test  Agency  Report  of  DoD  Instruction  5000.02 

Operational  Test  and  Evaluation 

Results 

Preliminary  Design  Review  Report  DoD  Instruction  5000.02 

Program  Protection  Plan  (for  DoD  Instruction  5200.39 

programs  with  critical  program 

information)  (includes  Anti- 

Tamper  Annex)  (also  summarized 

in  the  Acquisition  Strategy) 

Spectrum  Supportability  DoD  Directive  4650.1 

Determination  (applicable  to  all 
systems/equipment  that  use  the 
electromagnetic  spectrum  in  the 
U.S.  and  in  other  host  nations) 
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System  Threat  Assessment  Report 
(STAR) 

-  validated  by  Defense  Intelligence 
Agency  (DIA)  for  AC  AT  ID 
programs) 

-  validated  by  DoD  Components 
for  AC  AT  IC  programs 

-  Programs  on  the  DOT&E 
Oversight  List  require  a  STAR 
regardless  of  AC  AT  designation 

DoD  Instruction  5000.02 

DoD  Directive  5105.21 

DIA  Directive  5000.200 

DIA  Instruction  5000.002 

Systems  Engineering  Plan 

DoD  Instruction  5000.02 

Technology  Readiness  Assessment 

DoD  Instruction  5000.02 

Test  and  Evaluation  Master  Plan 

DoD  Instruction  5000.02 
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Appendix  G:  Questionnaire 


SAF/ACPO  &  The  Center  for 
Reengineering  &  Enabling 
Technology  (CRET) 


FROM;  SAF/ACPO- CRET 

SUBJECT :  Acquisition  Knowledge  Based  Decision  Support  (AKBDS) 

REF:  SAF/AQX  Memo  dated  18  Mar  09 

SAF/AQ  is  addressing  key  findings  from  GAO-08-467SP  ‘'Defense  Acquisitions, 
Assessments  of  Selected  Weapon  Programs"  which  calls  for  DoD  elements  to  develop  an 
improved  and  consistent  knowledge-based  decision  making  approach  within  the  acquisition 
community. 

A  team  led  by  the  CRET  has  been  established  to  review  and  validate  these  GAO  findings 
and  improve  the  fidelity  of  acquisition  decision  making  within  SAF/AQ  by  developing  and 
executing  a  disciplined  knowledge-based  approach  for  making  key  acquisition  decisions. 

The  OPR  for  this  effort  is  Mr.  John  Coakley  SAF/ACPO  (John.Coakley@pentagon.af.mil). 
He  can  be  directly  contacted  at  (703)  253-5617.  CRET  contacts:  Mr.  Anthony  Caruso 
(acaruso@dsdlabs.coin)  or  Mr.  Steve  Felosa  (sfelosa@dsdlabs.com).  Both  can  be  directly 
contacted  at  304-842-9870.  Please  direct  questions  to  the  above  individuals. 

As  POC  for  your  organization,  our  team  needs  collaborate  with  you  on  an  initial  45  -  60 
minute  teleconference  prior  to  22  April  09.  Please  respond  with  vour  preference  Oil 
date  and  time.  We  are  looking  for  your  support  to  assist  the  AKBDS  team  in  documenting 
your  organization's  actions  supporting  key  decisions  and  how  you  report  those  decisions.  This 
will  help  us  define  and  record  the  current  decision  making  process.  Capturing  this  data  will 
provide  the  foundation  of  a  disciplined,  knowledge-based  approach  for  decision  making  and 
information  sharing.  Assistance  for  this  phase  of  the  effort  will  include  participation  in  the  form 
of  surveys,  teleconferences,  and  minimal  face-to-face  meetings  through  June  2009. 

The  purpose  of  these  desired  collaborative  interactions  are  two-fold: 

(1)  To  identify  the  decision  products  that  your  organization  creates  and  submits  to  a 
customer.  This  customer  then  uses  the  product(s)  as  the  basis  or  a  tool  in  an  Acquisition 
Decision  (Review,  Event,  etc.). 

(2)  To  identify  the  decision  products  that  your  organization  receives  from  another 
organization.  Your  organization  then  uses  the  product(s)  as  a  tool  for  the  basis  of  an 
Acquisition  Decision  (Review.  Event,  etc.). 

To  help  us  in  our  discussions  -  the  below  questions  are  areas  we  would  like  to  discuss  with  you: 
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Does  your  organization  have  any  ongoing  initiatives  that  address  GAO  findings?  If 
so,  what  is  the  initiative  and  what  is  its  purpose? 


■  What  major  decision  product(s)  does  your  organization  create?  (A  decision  product  is 
defined  as  any  object  (report,  review,  plan,  document  package,  database  update,  etc.)  that  is  used 
to  make  an  Acquisition  Decision.  These  items  include  Budget  Execution  Documentation, 
Justification  Documents,  Monthly  Acquisition  Reports,  etc.  Items  that  we  are  not  looking  to 
identify  are  BURPS,  Weekly  Reports,  Org  Status  Reports,  etc.) 

■  Who  are  the  customer(s)  of  these  decision  product(s)? 

■  What  decision  product(s)  do  you  receive? 

■  What  are  the  source(s)  or  supplier(s)  of  these  products? 

■  What  feedback  does  your  organization  receive  from  your  customer(s)? 

■  What  feedback  does,  would  your  organization  give  to  your  suppliers(s)? 


■  What  acquisition  decision(s)  does  your  customer(s)  make  relative  to  the  decision 
product(s)  your  organization  provides  to  them?  (An  Acquisition  Decision  is  an  extent  that 
yields  an  outcome  that  has  an  impact  on  another  organization  or  ACAT  designated  program 
within  the  Acquisition  Lifecycle). 

■  What  acquisition  decision(s)  does  your  organization  make  relative  to  the  decision 
product(s)? 

■  What  is  the  intent  of  the  decision  product?  Does  the  decision  product  meet  its 

intended  purpose? 


■  What  information  products  do  you  believe  that  your  organization  should  receive 
(missing  information)  that  would  improve  your  acquisition  decision  making 
process? 

■  What  information  products  does  your  organization  receive  that  are  11011-value  added 
in  the  acquisition  decision  making  process? 

■  What  Regulations  and  Policies  require  that  your  organization  creates  these 
products? 

■  Do  you  use  an  IT  System  to  create,  retrieve  a  decision  product? 
If  so,  what  is  its  name? 
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